A scrolling LED Display for your Google Calendar!

My brother had the awesome idea to stay organized this school year with a scrolling LED display that would read the next five tasks off a Google Calendar. This is the exciting end result! The LED display we used unfortunately ran only proprietary software called PowerLed, giving me no way to programmatically set the text the screen was displaying and instead requiring each line be manually entered (like some kind of neanderthal). Fortunately Python has tools for automating actions in even the gnarliest GUI programs, so I wrote up this guide about how I managed to make this dumb device smart!

THE PROBLEM

It’s <current year> and manufacturers are still shipping hardware with awful proprietary software and no way for the end user to tinker with it. Additionally it is not web-friendly and all information has to be edited by hand.

The specific sign I worked with was this one although many signs originating in Asia ship with the same PowerLed software, so this guide should help you out! I made use of the WiFi interface, so if your sign does not have one you should check if it has a serial interface instead and make a few modifications to my program (remove the Wifi stuff, have the program click the button to send the configuration over USB for example).

THE SOLUTION

(TL;DR here is the code!)

There are a number of tools that let you click through GUI programs and automate tasks with Python, but today I am going to focus on pywinauto, a popular set of modules for automating tasks in Windows GUI programs. This is appropriate because as far as I can tell my sign only came with a Windows version of the software (although there are mobile apps that work with it supposedly). Pywinauto works by opening a program at a specified path and then navigating down menus it can see. For example, I can tell it to send my configuration by telling it to click a certain member of the Tools menu at the top of the screen: ‘window.menu_select(“Tools(T)->Send All”)’. It’s that easy! Those names were copied directly from the GUI itself.

I made use of the Google Calendar API which lets me programmatically pull down information from a Google Calendar. At the time of writing Google gives a generous one million queries per day which I cannot imagine one person would ever be able to exceed. To be safe though, my code updates the sign’s text every three minutes (although this can be changed at the top of main.py). The program I wrote pulls down the next five events on your calendar and processes them to be displayed by the sign.

How does this all work?

  • PowerLed uses ledprj files which are essentially just XML files. I made my own template with placeholder text (‘REPLACE1’, ‘REPLACE2’, etc.) and saved this template alongside my Python script. This template let me play with scroll speed, font size, and other settings using the PowerLed software and trial and error with my display. To automate updating the template with Python, I simply read in this template file and do a find/replace for ‘REPLACE1’ and so forth with the five events I pulled from Google Calendar. When PowerLed reads this file, these strings are sent instead!
  • The LED sign broadcasts its own WiFI network that I must connect to in order to update the device. I use subprocess calls to netsh to drop off my WiFi and then connect to the display.
  • I use pywinauto to open PowerLed, which I have already pointed at my LEDprj file, and connect to the device. I then automated the process of pressing the ‘send’ button to update the display. Once updated, pywinauto automatically closes PowerLed.
  • The project then waits three minutes and pulls down the next five events again, updates the sign, and so on until the program is stopped.

By noticing that the LEDPrj files were really just XML files I saved a great deal of time reverse engineering how I can get Python to update this sign. While there is no API or command line interface for the software, pywinauto let me quickly get the computer updating the sign itself.

Now I am sure you’re thinking ‘This is all fine and good, but do I need to let this program run forever, popping open this PowerLed thing every three minutes?’ – the answer is yes. But there is a solution! Upcycle an old laptop (68% of Americans have an unwanted computer in their home) or make use of a cheap Intel Compute Stick or similar device (make sure it runs Windows, the cheapest ones run Linux). I used an inexpensive Windows compute stick with an Intel Atom processor inside that more than handily can run this program on loop forever.

How do you get this set up for yourself? Head on over to the Git repo and check out my README, it’ll fill you in! The gist is to clone the repository, follow Google’s procedure to enable the Calendar API and get a credentials file, change the values in the program so that it knows what time zone you are in and what networks it needs to connect to (I don’t pass credentials, the program expects you to have connected to your sign and your home network before) and then give it a try!

TROUBLESHOOTING

  • I assume you have PowerLed installed and you made sure the path in the program is pointed at where you installed it
  • I assume you have connected to your sign and your home network before
  • I assume that you have loaded display.ledprj in PowerLed and then exited the program so that it will load next time you open PowerLed
  • I assume you are running Windows and that you have installed Python 3

SHARING THE SOLUTION

My code and instructions can be found here.

Here are the exact products I used:

How to extract PDF file attachments using Python and PyPDF2

Tl;dr: Cut and paste the function I wrote here.

This is a quick technical writeup to hopefully answer a question I’ve seen posted a few times around StackOverflow and the issue trackers of various Python PDF libraries. This is especially handy for those of you who don’t want to dive through the PDF32000 to figure out how Adobe wants us to handle attachments.

PyPDF2 makes working with PDFs easy, but you may have noticed that it only has an addAttachment() function, similar to many other PDF libraries I tried. How do we extract attachments so that we can work with them? Embedding files in PDFs is very common and it would be nice to be able to interact with these objects, like we can with form fields and other things you might find in PDF files.

Fortunately the building blocks how how to do this are already available in the PdfFileReader class!  We just need to stitch them together:

  1. Read the PDF file using PdfFileReader from PyPDF2
  2. Decrypt the PDF if necessary (required, you can’t get to the embedded files without doing this)
  3. Retrieve the file catalog by retrieving the file trailer (reader.trailer[‘/root’])
  4. Navigate in the dictionary this returns to ‘/EmbeddedFiles’
  5. Loop through the list of files that are found there
  6. When we get to an IndirectObject, we have our file parameters. We call getObject() to return the parameters dictionary, then navigate to ‘/F’ where our file data is stored as yet another IndirectObject. Here we simply call getData() and get a byte string back. This can then be written to a destination file or processed however you please!

As always it’s better to show the code, so here’s a proof of concept script:

Easy, just not immediately intuitive when you want to do this fast! I created pull request to hopefully get this function added as a method for the PdfFileReader class.

 

Use Python to collect and save data from microcontrollers to your PC

I’m at PyCon 2018 in Cleveland this upcoming week, so I thought that it would be a great time to post a quick Python writeup on how you can collect data from microcontrollers like Arduino and then save the data to your PC to process however you wish! PySerial makes this remarkably easy. This is a great project for science teachers as they show how you can use simple, ~$10 microcontrollers to create excellent data collection systems by connecting a few sensors and modifying my simple software! If you want to skip the tutorial you can download my software for Windows or Mac at the bottom of this page or use the link there to clone my repository on GitHub.

As usual,

THE PROBLEM

How can I collect a bunch of sensor data and then save it to my computer so that I can analyze it in a Jupyter notebook or in Excel?

THE SOLUTION

Over serial, and then use a simple Python program to save the data into a format like CSV that can be easily read by any analysis program!

For this writeup, I was specifically trying to collect data from an analog temperature sensor (TMP36) and then plot that temperature over time. I used a Teensy LC- a $12 Arduino-compatible microcontroller board that is so cheap and easy to use that I throw them at any of my rapid-prototyping problems. It features an ARM Cortex-M0+ processor, lots of analog and digital pins (including capacitive touch!) and a convenient micro-USB port for setting up an easy USB serial connection.

Here’s how I wired up the sensor:

temp_diagram.001
This probably would not pass UL certification.

Since all of my projects must include 3D printing, I also 3D printed a great case off of Thingiverse by a user named Kasm that nicely contains the project and makes sure I don’t accidentally short any of the pins to anything:

20180504_204414
Slightly more safe and professional looking. The temperature sensor is down on the bottom left.

For those curious how this works, the temperature sensor is an analog component that, when given 3-5V, returns a voltage that is proportional to the temperature around it. We are feeding the voltage into an analog to digital converter (ADC) so that our digital microcontroller can make sense of the reading. This means that in our code, we need to convert a number from 0-1023 back into a voltage so that we can use the formula from the data sheet to convert to a temperature. Fortunately, this is a simple conversion:

gif.latex

This will return a voltage in millivolts. The 3300 comes from the 3.3V reference voltage of the Teensy, and the 1024 comes from the Teensy having a 10-bit ADC, meaning it has 210 discrete analog levels.

We can then convert to a temperature using the formula from the device data sheet:

gif.latex-2

This will return a temperature in Celsius. I have adapted the Arduino sketch from this excellent Adafruit tutorial so that it will work with the Teensy:

Note what we are doing here:

  1. We are opening a serial port at 9600 baud (bits per second)
  2. We are reading from analog pin 9 on the Teensy, where the voltage out pin is connected
  3. We are converting the reading there to a voltage.
  4. We are converting that voltage to a temperature
  5. We are printing that line to serial so that it is transmitted
  6. We convert again from Celsius to Fahrenheit
  7. We print this line to serial as well
  8. We print a new line character so each row only has the one pair of readings
  9. We wait one minute to take another reading

Pretty simple! You can change the delay to any interval you wish to collect more or less data- the units are milliseconds. I wanted to only collect data once per minute, so I set it to 60000 milliseconds. Go ahead and flash this to your Teensy or Arduino using the Arduino IDE, making sure your device type is set to Serial. This is important, as without doing it you will not be able to read the data being sent over serial USB.

Entypo_e757(0)_1024.png
All software must have a nice icon.

Now to build the Python program. If you would rather simply download the binaries or clone the repository on Github for the Python client the links are available at the bottom of this page. We are going to use PySerial. We are also going to use Python 3, so make sure you have it installed if you are still using Python 2. Pip install PySerial by typing pip install pyserial. The logger is going to be wrapped in a simple Tkinter GUI. We want to accomplish the following:

  • Read the temperature data being printed to serial by the Teensy
  • Save it to a CSV value
  • Let the user select an interval to read from serial
  • Let the user select where they want the csv file saved
  • Let the user select what serial port they want to read off of.

The end result will look like this:

gui_screenshot
Who says Tkinter has to look ugly?

Let’s take a look at the code and then break it down:

What’s happening here is actually more simple than it seems! Let’s go through how it works, step by step, starting in Main():

  1. We create the root window where all of our widgets are stored (buttons, fields, etc.)
  2. We give it a name
  3. We grid a label
  4. We grid a dropdown menu
  5. We use PySerial to look at all serial ports open on the system and look for one called ‘USB Serial’. By default when you flash the Arduino or Teensy with the sketch shown above it will appear as ‘USB Serial’ on either Windows or Mac computers. We want to make sure the user selects the right serial port, so we filter the choices to these. More than likely there will be only one- the Teensy or Arduino connected to the machine.
  6. We populate the dropdown menu with these serial ports.
  7. We add a button and a counter. The default value of this counter is 60. This is letting the user know that they don’t have to sample every 60 seconds, and they can increment this value if they wish. You can change this value to whatever sampling rate you have set on your Teensy or Arduino in the sketch above.
  8. We add a button that lets the user start the data collection. This triggers a function where we ask the user where they want to save their output file in a save dialog, then open a serial port we called sp using PySerial. We must make sure we open it at 9600 baud, just like the serial port we opened on the Teensy! This is so the two ports can talk to each other. We then create a new thread to run our code in- if we did not, the program would freeze and you would not be able to stop the program because it would constantly be running the timer() function, collecting data, and never returning to the thread running the GUI! We set up this thread to run the timer() function as long as the global variable running is set to true. This lets us have a stop condition to terminate the thread when we are finished. In the timer function we use the filename the user has given to use and every sixty seconds save a time stamp in the first column, the temperature in Celsius in the second column, and the temperature in Fahrenheit in the third column. We read a line of of serial using the PySerial serial port’s readline() function and then convert whatever it gets to UTF-8 to make sure that what we are saving are readable Unicode characters. The output database is always being updated with each minute’s data in a new row! We open the CSV value in append mode (open(f.name, a)) so that we do not overwrite any existing data.
  9. We add a button to stop data collection. This sets the global variable running to false, causing the thread where we are saving the temperature data to stop. This halts all data collection.

And we are done! PySerial handles the heavy lifting and we have a nice, simple GUI to collect our data from. The CSV files can then be read into your favorite analysis program to create plots or fits:

Screen Shot 2018-05-05 at 12.28.30 PM
A quick plot I created in Numbers, showing that over the short interval tested the temperature did not stray far from 23 degrees.

SHARING THE SOLUTION

You can clone or fork the repository that contains all of these files here or you can download binaries to use right away on your system:

Clone the Repository

Download for macOS High Sierra or Windows 10