More plotting.

As I mentioned in a previous post, I was interested in data collection using an environment sensor, logging that info to a csv file and plotting a graph using gnuplot.

However, the incorrectly labelled graph axes annoyed me so I set about repairing them.

Also, I’m not going to revisit how it all sits together as that was done in the previous post .

In summary there is:

  • A python file that collects data from the Enviro pHAT (i.e. the enviroplot.py)
  • A .csv file that the python file writes comma separated values to (i.e. the enviroplot.csv)
  • The gnuplot script that interpret the .csv rows and produce the .png graphs
    • temp_pressure_last24hours.plot
    • temp_pressure_last7days.plot
    • temp_pressure_all.plot
  • An .sh file that wraps the python file and the gnuplot scripts together (i.e. enviroplot.sh)
  • A cron job in the Pi then executes the .sh file every 15 minutes (i.e. the pi crontab)
  • A cron job in the server then runs every 15 minutes to collect all the .png files (i.e. the server crontab)

The problem

The main issue with the graph axes was the underlying data in the .csv file (unsurprisingly).

I thought I was being clever and splitting out the date and time into separate columns, as it turns out that wasn’t so wise.

(I thought that later analysis such as ‘shown me the average temperatures by hour over time’ might be easier if I split these columns beforehand.)

The Python code would need a rewrite anyway to address my error but what about the existing info?

I had to choose from:

  • The quick way – abandon all the previously collected data and start afresh with the date and time captured correctly
  • The time consuming way – concatenate the existing columns into a format gnuplot could understand easier

Obviously I chose the time consuming way and fired up Excel.

(I also took the opportunity to rewrite the plotting scripts as I had discovered that gnuplot was capable of creating dual axes graphs.)

The data collection bit

The Python code that writes to the .csv file now looks like this:

enviroplot.py 

#!/usr/bin/env python

import csv
from datetime import datetime
from envirophat import weather

dt = datetime.now()
DateTime= dt.strftime('%d-%m-%Y %H:%M:%S')
temp = round(weather.temperature(), 2)
baro = round(weather.pressure(unit = 'hPa'), 2)

csvWrite = DateTime, temp, baro
csvFile = open('/home/foo/Dev/RasPi/enviroplot/enviroplot.csv', 'a')

with csvFile:
        writer = csv.writer(csvFile)
        writer.writerow(csvWrite)

The created .csv file now looks like this, with a single column for Date and Time:

The logging bit

enviroplot.csv (Example of data collected)

DateTime temp baro
23-01-2019 22:15:01 23.12 998.17
23-01-2019 22:30:01 22.89 998.35
23-01-2019 22:45:02 22.54 998.41
23-01-2019 23:00:02 22.2 998.44
23-01-2019 23:15:01 21.9 998.46

The plotting part

The gnuplot script that creates a 640×480 dual axes graph in .png format of temperature and pressure over the last 24 hours.

temp_pressure_last24hours.plot 

# Create new plot
reset
unset key 
set key off 
set terminal png medium size 640,480

# csv DateTime format
set datafile separator ","
set timefmt "%d-%m-%Y %H:%M:%S"

# Graph formatting
set title "Temperature and Pressure Over the Last 24 Hours"
# set grid

# x axis formatting
set xdata time
set format x "%l%p"

# y axis formatting
set ytics 1 nomirror tc lt 1
set ylabel "Temperature" tc lt 1

# y2 axis formatting
set y2tics 1 nomirror tc lt 2
set y2label "Pressure" tc lt 2

# Plot the data
plot '< tail -n 90 /home/foo/Dev/RasPi/enviroplot/enviroplot.csv' using 1:2 axes x1y1, '< tail -n 90 /home/foo/Dev/RasPi/enviroplot/enviroplot.csv' using 1:3 axes x1y2 with lines

The gnuplot script that creates a 640×480 dual axes graph in .png format of temperature and pressure over the last 7 days.

temp_pressure_last7days.plot 

# Create new plot
reset
unset key 
set key off 
set terminal png medium size 640,480

# csv DateTime format
set datafile separator ","
set timefmt "%d-%m-%Y %H:%M:%S"

# Graph formatting
set title "Temperature and Pressure Over the Last 7 Days"
# set grid

# x axis formatting
set xdata time
set format x "%a"

# y axis formatting
set ytics 1 nomirror tc lt 1
set ylabel "Temperature" tc lt 1

# y2 axis formatting
set y2tics 5 nomirror tc lt 2
set y2label "Pressure" tc lt 2

# Plot the data
plot '< tail -n 672 /home/foo/Dev/RasPi/enviroplot/enviroplot.csv' using 1:2 axes x1y1, '< tail -n 672 /home/foo/Dev/RasPi/enviroplot/enviroplot.csv' using 1:3 axes x1y2 with lines

The gnuplot script that creates a 640×480 dual axes graph in .png format of temperature and pressure over all recorded data.

temp_pressure_all.plot 

# Create new plot
reset
unset key 
set key off 
set terminal png medium size 1280,480

# csv DateTime format
set datafile separator ","
set timefmt "%d-%m-%Y %H:%M:%S"

# Graph formatting
set title "Temperature and Pressure Over All Time"
# set grid

# x axis formatting
set xdata time
set format x "%b %y"

# y axis formatting
set ytics 10 nomirror tc lt 1
set ylabel "Temperature" tc lt 1

# y2 axis formatting
set y2tics 10 nomirror tc lt 2
set y2label "Pressure" tc lt 2

# Plot the data
plot '/home/foo/Dev/RasPi/enviroplot/enviroplot.csv' using 1:2 axes x1y1, '/home/foo/Dev/RasPi/enviroplot/enviroplot.csv' using 1:3 axes x1y2 with lines

Automating using cron

The .sh file to pull together the enviroplot.py and gnuplot scripts.

enviroplot.sh 

#!/bin/sh
# This is the script that the cron job will execute

# Run the python script to collect the variables
python /home/foo/Dev/RasPi/enviroplot/enviroplot.py

#### Create the graph images from gnuplot
# 'Sleep' is there to prevent a conflict
sleep 10
gnuplot /home/foo/Dev/RasPi/enviroplot/temp_pressure_all.plot > /home/foo/Dev/RasPi/enviroplot/temp_pressure_all.png
sleep 10
gnuplot /home/foo/Dev/RasPi/enviroplot/temp_pressure_last7days.plot > /home/foo/Dev/RasPi/enviroplot/temp_pressure_last7days.png
sleep 10
gnuplot /home/foo/Dev/RasPi/enviroplot/temp_pressure_last24hours.plot > /home/foo/Dev/RasPi/enviroplot/temp_pressure_last24hours.png

The pi crontab that executes every 15 minutes to run enviroplot.sh.

pi crontab

# This is the script that runs the bash script to collect data and create graphs
*/15 * * * * /home/foo/Dev/Raspi/enviroplot/enviroplot.sh

The cron job on the server has changed slightly as it now pulls in all of the .png files and not the named ones as per the previous version.

server crontab

# Collect the graphs from the 'piboard' machine
*/15 * * * * scp -i ~/.ssh/id_rsa_piboard foo@piboard:/home/foo/Dev/RasPi/enviroplot/*.png /var/www/html/

And finally!

This is what the output looks like, snipped from a web page.

Now that the visible range of data displayed has been shortened you can see that the temperature rises most evenings (when the heating comes on) and the following days weather can be predicted by looking at the pressure graphs.

Data collection using an enviroment sensor, logging to a csv file and plotting a graph using gnuplot. Screen-grab of two graphs side by side showing the temperature and pressure. The left hand graph shows temperature and pressure over the last 24 hours, while the right hand one shows temperature and pressure over the prior 7 days. Temperature and Pressure Over the Last 24 Hours and Prior 7 Days

All of the Enviro pHAT code examples can also be found on Forgejo .