The good kind of plotting

I was interested in data collection using an environment sensor, logging that info to a csv file and plotting a graph using gnuplot.

I have a couple of RaspberryPi HATs and pHATs knocking about and wondered how to collect data from one, and log the data it to a local file and then present that info in an understandable way.

The routine below should be a good jumping off point for people to start with and does the following:

  • Uses python to collect data from a sensor (in my case an Enviro pHAT)
  • Then, writes the python output to a comma separated file
  • Uses gnuplot to turn slices of the .csv file in to graphs, stored locally
  • Instructs a web server on a different machine to pull those files into the /var/www folder

A web page then easily serves these, as they always have the same name.

So, putting all of this together:

The data collection bit

The Enviro pHAT sits on top of a pi Zero WH (named phats). It uses ssh keys for authentication.

The pi is connected to the internal network (and therefore the web server) and has a known IP address.

The web server is named sheevaplug, it has a single user with no rights allocated. The user has to su to get rights.

On the pi there are a few files:

  • The python file that collects data from the Enviro pHAT (enviroplot.py)
  • The .csv file that the python file writes comma separated values to (enviroplot.csv)
  • The gnuplot script(s) that interpret the .csv rows and produce the .png graphs (temp_range.plot, temp_recent.plot)
  • The sh file that wraps the python file and the gnuplot scripts together (enviroplot.sh)
  • A cron job then executes the sh file every 15 minutes

The logging bit

enviroplot.py

#!/usr/bin/env python

import csv
from datetime import datetime
from envirophat import weather

dt = datetime.now()

full_date = dt.strftime(’%Y-%m-%d’)
full_time = dt.strftime(’%H:%M:%S’)
full_day = dt.strftime(’%A’)
year = dt.year
month = dt.month
date = dt.day
hour = dt.hour

temp = round(weather.temperature(), 2)
baro = round(weather.pressure(unit = ‘hPa’), 2)

csvWrite = full_date, full_time, full_day, year, month, date, hour, temp, baro
csvFile = open(’/home/foo/Dev/Raspi/enviroplot/enviroplot.csv’, ‘a’)

with csvFile:
writer = csv.writer(csvFile)
writer.writerow(csvWrite)

The important bits here are:

  • round(xxxx, 2) rounds to two decimal places
  • The ‘a’ at the end of the csvFile line means append. The .csv file must exist first, even if it is empty to start with

The logging output

enviroplot.csv (Example of data collected)

full_date, full_time, full_day, year, month, date, hour, temp, baro
2019-01-23,22:15:01,Wednesday,2019,1,23,22,23.12,998.17
2019-01-23,22:30:01,Wednesday,2019,1,23,22,22.89,998.35
2019-01-23,22:45:02,Wednesday,2019,1,23,22,22.54,998.41
2019-01-23,23:00:02,Wednesday,2019,1,23,23,22.2,998.44
2019-01-23,23:15:01,Wednesday,2019,1,23,23,21.9,998.46
2019-01-23,23:30:01,Wednesday,2019,1,23,23,21.73,998.82
2019-01-23,23:45:01,Wednesday,2019,1,23,23,21.47,999.24
2019-01-24,00:00:01,Thursday,2019,1,24,0,21.18,999.26
2019-01-24,00:15:01,Thursday,2019,1,24,0,20.96,999.3
2019-01-24,00:30:02,Thursday,2019,1,24,0,20.92,999.32
2019-01-24,00:45:01,Thursday,2019,1,24,0,20.87,999.64
2019-01-24,01:00:01,Thursday,2019,1,24,1,20.63,999.82

I have deliberately collected the full date, full time, day, year, month, date, hour separately as I may want to analyse these elements later and don’t want to have to dissect the .csv string.

(The final two values are temperature and barometric pressure.)

The plotting part

The following two .plot examples are the gnuplot scripts.

Gnuplot is an incredibly versatile open source plotting piece of software and is available in the repos.

sudo apt install gnuplot

It is capable of creating 2D and 3D graphs and outputting to a variety of formats.

You can even plot ASCII graphs in the terminal with it!

You can create a graph with as little as one line, specifying simply where the input data is.

I’m still learning, so you may have better looking outputs with more practice.

temp_range.plot

reset

unset key
set key off

set terminal png

set datafile separator ","

set xdata time
set timefmt "%Y-%m-%dT%H:%M:%S"
set format x "%d/%m"

set xlabel "Date (day/month)"
set ylabel "Temp"

set style data points

set title "Temp Range"
set grid

plot '/home/foo/Dev/Raspi/enviroplot/enviroplot.csv' using 1:8

The important bits here are:

  • ‘plot’ i.e. the location of the .csv file containing data
  • ‘Using 1:8’ i.e. use data columns 1 & 8

temp_recent.plot

reset

unset key
set key off

set terminal png

set datafile separator ","

set xdata time
set timefmt "%Y-%m-%dT%H:%M:%S"
set format x "%d/%m"

set xlabel "Date (day/month)"
set ylabel "Temp"

set style data line

set title "Recent Temp"
set grid

plot '< tail -n 672 /home/foo/Dev/Raspi/enviroplot/enviroplot.csv' using 1&2:8

The important bits here are:

  • ‘tail -n 672’ i.e. take the last 672 data points ( 1 x week at 15 minute intervals)
  • ‘using 1&2:8’ i.e. concatenate columns 1&2 as one data column and column 8 as the other

Automating using cron

enviroplot.sh

#!/bin/sh
# This is the script that the cron job will execute

# Run the python script to collect the variables
python /home/foo/Dev/Raspi/enviroplot/enviroplot.py

#### Create the graph images from gnuplot
# 'Sleep' is there to prevent a conflict
gnuplot /home/foo/Dev/Raspi/enviroplot/temp_range.plot > /home/foo/Dev/Raspi/enviroplot/temp_range.png
sleep 5
gnuplot /home/foo/Dev/Raspi/enviroplot/temp_recent.plot > /home/foo/Dev/Raspi/enviroplot/temp_recent.png

The important bits here are:

  • Runs the python script that writes the Enviro pHAT data to the .csv file
  • Calls the gnuplot file(s) and outputs it to a named .png file i.e. creates the graph image file(s)
  • ‘Sleep’ is there as cron will execute both commands at the same time; better to have a slight delay and wait for the first command to finish

phats crontab

# m h dom mon dow command
# This is the script that runs the bash script to collect data and create graphs
*/15 * * * * /home/foo/Dev/Raspi/enviroplot/enviroplot.sh

The cron job that runs every 15 minutes to execute the above enviroplot.sh script.

The above will get you a working pair of graphs that are refreshed every 15 minutes.

But they will be on the pi which may be headless or unattached to any sort of display.

Much better to put the images on the interweb where you can watch your house getting hotter when it’s burning.

The next two files are on the web server and comprise of:

  • A sh script to pull the files from the pi over the local network
  • A cron job to do that every 15 minutes

Serving it all up

I have to pull the files into the web server, rather than push the files from the pi because, as I said earlier the server only has a single user with no rights and I couldn’t find a way to connect to the server using ssh, then pass a password to get rights. (Probably is a way, but too late now.)

get_graphs.sh

#!/bin/sh
# This is the script that the cron job will execute

# Collect the graphs from the 'phats' machine
# 'Sleep' is there to prevent a conflict

scp -i ~/.ssh/id_rsa_phats foo@phats:/home/foo/Dev/Raspi/enviroplot/temp_range.png /var/www/html/
sleep 5
scp -i ~/.ssh/id_rsa_phats foo@phats:/home/foo/Dev/Raspi/enviroplot/temp_recent.png /var/www/html/

The important bits here are:

  • scp is secure copy and allows use of ssh keys
  • Also, I’m using ssh keys for authentication, this prevents having to type my password repeatedly and offers secure copying and logins
  • ‘Sleep’ is there as cron will execute both commands at the same time; better to have a slight delay and wait for the first command to finish

sheevaplug crontab

# m h dom mon dow command
# This is the script that runs the bash script to collect the graphs from the 'phats' machine
*/15 * * * * sh /var/www/html/get_graphs.sh

The cron job that runs every 15 minutes to execute the above get_graphs.sh script.

And finally!

This is what the output looks like, snipped from a web page.

Data collection using an enviroment sensor, logging to a csv file and plotting a graph using gnuplot. Screen-grab of two graphs side by side showing the temperature. The left hand graph shows a two week time period whilst the right hand one shows temperature over all time. The date axis on both is odd looking and clearly wrong. Temperature Over the Last Fortnight and All Time

Also, I know the axis on the first graph looks a bit odd, as I said I’m just starting out with gnuplot.

This method has now been updated , with corrected graphs axes on a dual axis chart, showing temperature and pressure over both the last week and the prior 24 hours.