Use Python to collect and save data from microcontrollers to your PC

I’m at PyCon 2018 in Cleveland this upcoming week, so I thought that it would be a great time to post a quick Python writeup on how you can collect data from microcontrollers like Arduino and then save the data to your PC to process however you wish! PySerial makes this remarkably easy. This is a great project for science teachers as they show how you can use simple, ~$10 microcontrollers to create excellent data collection systems by connecting a few sensors and modifying my simple software! If you want to skip the tutorial you can download my software for Windows or Mac at the bottom of this page or use the link there to clone my repository on GitHub.

As usual,

THE PROBLEM

How can I collect a bunch of sensor data and then save it to my computer so that I can analyze it in a Jupyter notebook or in Excel?

THE SOLUTION

Over serial, and then use a simple Python program to save the data into a format like CSV that can be easily read by any analysis program!

For this writeup, I was specifically trying to collect data from an analog temperature sensor (TMP36) and then plot that temperature over time. I used a Teensy LC- a $12 Arduino-compatible microcontroller board that is so cheap and easy to use that I throw them at any of my rapid-prototyping problems. It features an ARM Cortex-M0+ processor, lots of analog and digital pins (including capacitive touch!) and a convenient micro-USB port for setting up an easy USB serial connection.

Here’s how I wired up the sensor:

temp_diagram.001
This probably would not pass UL certification.

Since all of my projects must include 3D printing, I also 3D printed a great case off of Thingiverse by a user named Kasm that nicely contains the project and makes sure I don’t accidentally short any of the pins to anything:

20180504_204414
Slightly more safe and professional looking. The temperature sensor is down on the bottom left.

For those curious how this works, the temperature sensor is an analog component that, when given 3-5V, returns a voltage that is proportional to the temperature around it. We are feeding the voltage into an analog to digital converter (ADC) so that our digital microcontroller can make sense of the reading. This means that in our code, we need to convert a number from 0-1023 back into a voltage so that we can use the formula from the data sheet to convert to a temperature. Fortunately, this is a simple conversion:

gif.latex

This will return a voltage in millivolts. The 3300 comes from the 3.3V reference voltage of the Teensy, and the 1024 comes from the Teensy having a 10-bit ADC, meaning it has 210 discrete analog levels.

We can then convert to a temperature using the formula from the device data sheet:

gif.latex-2

This will return a temperature in Celsius. I have adapted the Arduino sketch from this excellent Adafruit tutorial so that it will work with the Teensy:

void setup()
{
Serial.begin(9600); //Start the serial connection with the computer
//to view the result open the serial monitor
}
void loop()
{
int reading = analogRead(9);
// Teensy reference voltage is 3.3V
float voltage = reading * 3.3;
voltage /= 1024.0;
// now print out the temperature
float temperatureC = (voltage – 0.5) * 100 ; //converting from 10 mv per degree wit 500 mV offset
//to degrees ((voltage – 500mV) times 100)
Serial.print(temperatureC);
// Print a space
Serial.print(' ');
// now convert to Fahrenheit
float temperatureF = (temperatureC * 9.0 / 5.0) + 32.0;
Serial.print(temperatureF);
// New line
Serial.print('\n');
delay(60000); //waiting a minute
}

Note what we are doing here:

  1. We are opening a serial port at 9600 baud (bits per second)
  2. We are reading from analog pin 9 on the Teensy, where the voltage out pin is connected
  3. We are converting the reading there to a voltage.
  4. We are converting that voltage to a temperature
  5. We are printing that line to serial so that it is transmitted
  6. We convert again from Celsius to Fahrenheit
  7. We print this line to serial as well
  8. We print a new line character so each row only has the one pair of readings
  9. We wait one minute to take another reading

Pretty simple! You can change the delay to any interval you wish to collect more or less data- the units are milliseconds. I wanted to only collect data once per minute, so I set it to 60000 milliseconds. Go ahead and flash this to your Teensy or Arduino using the Arduino IDE, making sure your device type is set to Serial. This is important, as without doing it you will not be able to read the data being sent over serial USB.

Entypo_e757(0)_1024.png
All software must have a nice icon.

Now to build the Python program. If you would rather simply download the binaries or clone the repository on Github for the Python client the links are available at the bottom of this page. We are going to use PySerial. We are also going to use Python 3, so make sure you have it installed if you are still using Python 2. Pip install PySerial by typing pip install pyserial. The logger is going to be wrapped in a simple Tkinter GUI. We want to accomplish the following:

  • Read the temperature data being printed to serial by the Teensy
  • Save it to a CSV value
  • Let the user select an interval to read from serial
  • Let the user select where they want the csv file saved
  • Let the user select what serial port they want to read off of.

The end result will look like this:

gui_screenshot
Who says Tkinter has to look ugly?

Let’s take a look at the code and then break it down:

#!/usr/bin/env python
import os
import serial
import serial.tools.list_ports
import threading
import tkinter as tk
import time
from threading import Thread
from time import sleep
from tkinter import filedialog as fd
from tkinter import ttk
running = True
f = ""
def timer(sp, interval):
global f
while running:
sleep(interval)
line = sp.readline().decode("utf-8")
data = [float(val) for val in line.split(' ')]
t = time.strftime("%Y-%m-%d %H:%M:%S")
newRow = "%s,%s,%s\n" % (t, data[0], data[1])
with open(f.name, "a") as datafile:
datafile.write(newRow)
def collect(strPort, interval):
global f
# Ask for a location to save the CSV file
f = fd.asksaveasfile(mode='w', defaultextension=".csv")
if f is None: # User canceled save dialog
return
# Overwrite existing file
try:
os.remove(f)
except:
# File does not exist yet
pass
sp = serial.Serial(strPort, 9600)
time_thread = Thread(target=timer, args=(sp, interval))
time_thread.start()
def end():
global running
running = False
def onIncrement(counter):
counter.set(counter.get() + 1)
def main():
root = tk.Tk()
root.title("Serial USB Temperature Data Collector")
mainframe = ttk.Frame(root)
mainframe.grid(column=0, row=0, sticky=(tk.N,tk.W,tk.E,tk.S))
mainframe.pack()
serial_label = ttk.Label(mainframe, text="Sensor Serial Port:")
serial_label.grid(row=0, column=0)
serial_var = tk.StringVar(root)
raw_ports = list(serial.tools.list_ports.comports())
ports = []
for p in raw_ports:
if "USB Serial" in p.description:
ports.append(p.device)
serial_menu = ttk.OptionMenu(mainframe, serial_var, *ports)
serial_menu.grid(row=0, column=1)
counter = tk.IntVar()
counter.set(60)
duration_label = ttk.Label(mainframe, textvariable=counter)
duration_label.grid(row=1, column=1)
duration_increment = ttk.Button(mainframe,
text="Increase Collection Interval (sec)",
command=lambda: onIncrement(counter))
duration_increment.grid(row=1, column=0)
collect_button = ttk.Button(mainframe, text="Begin Data Collection",
command=lambda: collect(serial_var.get(),
counter.get()))
collect_button.grid(row=2, column=0, sticky=(tk.E, tk.W))
end_button = ttk.Button(mainframe, text="End Data Collection",
command=lambda: end())
end_button.grid(row=2, column=1, sticky=(tk.E, tk.W))
root.mainloop()
if __name__ == '__main__':
main()

view raw
main.py
hosted with ❤ by GitHub

What’s happening here is actually more simple than it seems! Let’s go through how it works, step by step, starting in Main():

  1. We create the root window where all of our widgets are stored (buttons, fields, etc.)
  2. We give it a name
  3. We grid a label
  4. We grid a dropdown menu
  5. We use PySerial to look at all serial ports open on the system and look for one called ‘USB Serial’. By default when you flash the Arduino or Teensy with the sketch shown above it will appear as ‘USB Serial’ on either Windows or Mac computers. We want to make sure the user selects the right serial port, so we filter the choices to these. More than likely there will be only one- the Teensy or Arduino connected to the machine.
  6. We populate the dropdown menu with these serial ports.
  7. We add a button and a counter. The default value of this counter is 60. This is letting the user know that they don’t have to sample every 60 seconds, and they can increment this value if they wish. You can change this value to whatever sampling rate you have set on your Teensy or Arduino in the sketch above.
  8. We add a button that lets the user start the data collection. This triggers a function where we ask the user where they want to save their output file in a save dialog, then open a serial port we called sp using PySerial. We must make sure we open it at 9600 baud, just like the serial port we opened on the Teensy! This is so the two ports can talk to each other. We then create a new thread to run our code in- if we did not, the program would freeze and you would not be able to stop the program because it would constantly be running the timer() function, collecting data, and never returning to the thread running the GUI! We set up this thread to run the timer() function as long as the global variable running is set to true. This lets us have a stop condition to terminate the thread when we are finished. In the timer function we use the filename the user has given to use and every sixty seconds save a time stamp in the first column, the temperature in Celsius in the second column, and the temperature in Fahrenheit in the third column. We read a line of of serial using the PySerial serial port’s readline() function and then convert whatever it gets to UTF-8 to make sure that what we are saving are readable Unicode characters. The output database is always being updated with each minute’s data in a new row! We open the CSV value in append mode (open(f.name, a)) so that we do not overwrite any existing data.
  9. We add a button to stop data collection. This sets the global variable running to false, causing the thread where we are saving the temperature data to stop. This halts all data collection.

And we are done! PySerial handles the heavy lifting and we have a nice, simple GUI to collect our data from. The CSV files can then be read into your favorite analysis program to create plots or fits:

Screen Shot 2018-05-05 at 12.28.30 PM
A quick plot I created in Numbers, showing that over the short interval tested the temperature did not stray far from 23 degrees.

SHARING THE SOLUTION

You can clone or fork the repository that contains all of these files here or you can download binaries to use right away on your system:

Clone the Repository

Download for macOS High Sierra or Windows 10

Exploring 3D Printing with First Graders and Tinkercad

This month I have had the privilege to work with a class of talented first graders in order to teach them about computer aided design, or CAD. This is an essential tool in any engineer’s toolbox, but it using it can get very complicated very quickly. Frequent readers of this blog likely know my fondness for OnShape and it’s spiritual predecessor and industry titan, SolidWorks. These programs are far too complicated for a quick introduction in how to use CAD, and the number of features can be overwhelming. While OnShape does work on iPads, which these children are fortunate to have in their classrooms, there is a much friendlier way to introduce 3D design to children- Tinkercad.

Screen Shot 2018-02-23 at 10.16.35 AM.png
The Tinkercad interface and a quick elephant I drew using the basic primitives it offers.

Users create their models out of simple 3D shapes like boxes, cylinders, and pyramids just like they were building with wooden blocks. The big difference is that these blocks can be morphed and stretched like clay, letting users make pretty much whatever they want. They can set how big they are in millimeters and give a precise orientation in degrees.

Tinkercad is maintained by industry heavyweight Autodesk, which is famous for it’s CAD packages like AutoCAD and 3D art software like Maya and 3DS Max. It gives a nice set of features for beginners to computer aided design- lots of primitives and shape generators to choose from, rulers and workplanes to reference your work to and measure distances, the ability to dimension shapes, and a navigation system straight out of the best parametric modelers are all present here. This is great because it means the skills you develop learning Tinkercad transfer into using other popular packages, like OnShape or Inventor, likely the most similar Autodesk offering. As an engineer this is the biggest selling point for teaching children with this package- I know that whatever they get out of this activity, they can transfer forward to bigger and better things. This cannot be said for other programs I tried- Autodesk has another offering called 123D Sculpt that simulates making something out of clay that I found too limiting and too application specific. Microsoft Paint 3D is an impressive upgrade of the drawing program I remember, but it is more oriented towards 3D art than serious 3D part design.

That said, the activity we tackled was not one of designing parts for a rocket or model car but instead making models of the animals they were studying in their science unit. This may seem like a bad application on the surface for modeling programs like Tinkercad if the goal is to teach students how to make useful parts on 3D printers, but in my opinion it is perfect. A big part of imaging how a part should look or be designed is to imagine how it can be made out of simple shapes and figures. When working with a parametric modeler you often observe a designer create the basic shape out of circles, boxes, arcs, and lines and then begin to add the finer details. Working with something like an animal the children are familiar with is perfect for teaching this higher-level skill. It is one thing to teach them how to draw boxes and spheres and dimension them, but to teach them how to think about what they are designing and introducing them to the creative process is much more valuable than the technical skill of using the program itself. Take this elephant the children drew, for example:

Funky Jaagub-Esboo.png
An impressive elephant the first graders drew. This was the first model they created.

I had shown them my demo elephant, and they used it as a springboard to make their own. They identified the shapes I had used to make mine and impressed me with how they understood the creative process I had gone through to create those shapes. They then extended it by thinking of what shape they wanted to use for standing legs (I modeled mine laying down to make things easier), and what shape they wanted to use for eyes, both of which I did not have in my model! They also creatively added a baseplate for the elephant to be printed on with a label, taking advantage of the Tinkercad text tool. They were already thinking in terms of what shapes they had available to them and how they could be stretched and morphed into the figures they wanted. They were also thinking in terms of what could actually be 3D printed. These students had the privilege of having a 3D printer in the classroom and with a little coaching on my part realized why you cannot have big pieces hanging in thin air if you want a successful print, and identified that parts have to be stacked one on top of the other, not hanging in space. I taught them the ‘sandcastle principle’- imagine that you are making this model in sand. You cannot have sand floating in midair, and it won’t stay up without support. They seemed to quickly grasp the idea and they ran with it.

Here are a few of their amazing creations:

If you would like to view the presentation I gave to the students that explains some easy dos and don’ts about 3D printing and design, I’ve included it here in Google Slides format:

This was an awesome experience, and I know the students enjoyed it too. They are now obsessed with Tinkercad and 3D printing, and I have been amazed by what they have independently been creating. This is truly an awesome tool for introducing 3D art, design, and 3D printing to primary and intermediate students, as there is a lot of flexibility here for making very complicated things. I have even found that for very quick designs Tinkercad is more than adequate and it is fast. I am excited to return and work with a new set of students on another set of animals. I am sure I will be just as surprised at their creativity and inventiveness.

 

Making a Star Wars Poster Lithopane Lamp

Star Wars has some of the most iconic artwork of any movie franchise in history. Nowhere is this more apparent than in its theatrical posters. This lithopone lamp captures all nine main storyline Star Wars posters (with a blank standing in for Episode 9 until the official poster is released!) in a lovely desktop lamp that celebrates your favorite movies. Simply place an approximately 2.5 inch diameter LED puck light (like you would place under a cabinet) inside the indentation in the base and run the cord out the slot!

As always, you can edit this project on OnShape!

Also as usual you can download and print these files right now from my Thingiverse account.