Gus

_images/hamster.jpg

About

Gus, a Raspberry Pi 3 or 4 running Bullseye OS, is able to store and visualize not only SnifferBuddy readings, but also vpd values.

https://docs.google.com/drawings/d/e/2PACX-1vTjks0iZHIZyD4VEdOo01_se0jn_CgJu9JUCee-rUhXBmFfykmObBkpqSUFBkOvnIdisiIzygPvDeZa/pub?w=984&h=474&align=middle

As you can see, Gus has a lot of tools to help GrowBuddies. These include a mosquitto mqtt broker that holds the latest SnifferBuddy mqtt message. If instructed to do so, Gus can store the SnifferBuddy readings in an influxdb database. If the readings are stored, you no longer need Gus to access the readings. For example, you can create simple to sophisticated dashboards using Grafana. Or look at the data through apps that speak to influxdb’s API.

Gus is equipped with a range of tools to assist GrowBuddies, including a mosquitto mqtt broker, which holds the latest SnifferBuddy mqtt message. Additionally, Gus can store SnifferBuddy readings in an influxdb database upon request. This allows for the creation of dashboards using Grafana or access to the data through apps that utilize influxdb’s API, eliminating the need for ongoing access to Gus for reading purposes.

Let’s Make One

Comments & Questions

To ensure that Gus is able to handle the demands of his tasks, he is equipped with a robust set of tools. The process of setting up Gus begins with constructing his hardware, followed by installing the operating system and necessary tools to bring him to life.

Here are two I’ve made:

_images/Gus_2.jpg

The hardware for the green device (on the left) consists of components from the Raspberry Pi 4, while the purple device is built with components from the Raspberry Pi 3B+.

Rasp Pi + Enclosure

Gather The Materials

First, you’ll need to gather the materials.

  • A Raspberry Pi 3B+ or 4. At the time of this writing, there is a shortage of Raspberry Pis. I have had the best luck with Adafruit.

  • A Power Source for the Raspberry Pi. Note: the Raspberry Pi 4 (5V via USB type-C up to 3A) uses a different power supply than the Raspberry Pi 3 (5V via micro USB up to 2.5A).

  • A [microSD card with full-size adapter.

  • An Enclosure needs to be printed out on a 3D printer. The model I chose is [Malolo’s screw-less/snap-fit Raspberry Pi 3 and 4 cases](https://www.thingiverse.com/thing:3723561). Specifically the one-color slot base and the two-color hex top. You can choose what you want. I have included the stl files I used.

Put the Pi in its Enclosure

I found it pretty simple to put the pi into the enclosure once it had been printed.

Install The Software

Resources

I found the following stuff on the web to be helpful:

Install The Raspberry Pi OS

_images/rasppi_lite_install.jpg

Rasp Pi Lite OS

  • Choose the SD card, then go into Settings by clicking on the cog.

_images/rasppi_lite_install_2.jpg

Choose the cog

  • Enter gus for the hostname and enable SSH with password authentication.

  • Fill in the other options - wifi, username/password, wifi, and local settings.

_images/rasppi_lite_install_3.jpg

Setup Options

  • Click Save to install the Rasp Pi OS Lite OS with the options you entered.

  • Remove the SD card from the reader.

  • Insert the microSD into the Raspberry Pi.

Verify the Install

Go to a terminal window on your Mac or PC and type:

ssh pi@gus

After entering your password, you should be in the command prompt:

pi@gus:~ $

If you cannot reach Gus from your Mac/PC, first check to see if the raspberry pi is on your home wifi by using a utility like Angry IP. If it is not, perhaps this troubleshooting guide helps.

Install mqtt

The buddies use mqtt to send and receive payloads and commands. For example, SnifferBuddy sends out (i.e.: publishes in mqtt terminology) over wifi an mqtt message like this:

{"Time":"2022-09-06T08:52:59",
  "ANALOG":{"A0":542},
  "SCD30":{"CarbonDioxide":814,"eCO2":787,"Temperature":71.8,"Humidity":61.6,"DewPoint":57.9},"TempUnit":"F"}
}

Gus runs the Mosquitto Broker. Mosquitto is a lightweight open source message broker. It works well. Thank you to the open-source community.

Configure the Mosquitto MQQT Broker

Before installing the service, some unique settings are needed in Mosquitto’s config file.

  • Create the connect.conf file. From a terminal open on the Raspberry Pi:

    • $ cd /etc/mosquitto/conf.d

    • $ sudo nano connect.conf

    • copy/ paste the following into the new connect.conf file:

listener 1883
protocol mqtt

listener 9000
protocol websockets

allow_anonymous true
  • save and exit.

Install the Mosquitto MQQT Broker
Observe Messages

Open up MQTT explorer and connect to gus.

_images/mqtt_explorer.jpg

Gus mqtt broker

The image points out there is a SnifferBuddy sending mqtt messages to Gus.

Using MQTT To Determine Device Health

You may not need to know anything about this. I have it here so I don’t forget why this is in the code!

MQTT’s Last will and Testament - LWT is an extremely useful feature to use for mqtt error debugging. The Buddies sending mqtt messages are all Tasmota devices. Refer to Tasmota’s page for advice on how LWT is implemented.

For example, let’s say we have a SnifferBuddy up and running.

pi@gus:~ $ mosquitto_sub -t "tele/snifferbuddy/LWT"
Online
Offline
Online

subscribing to the above topic shows initially SnifferBuddy is online. I unplug. Wait the 30 seconds that Tasmota by default sets the keep-alive time to. An offline message is sent by the broker. I then plug SnifferBuddy back in. Right after SnifferBuddy boots, we receive an Online message.

The MQTT Esentials Page 9 article lists several states that cause the broker to publish the LWT message.

issuing the status 6 command on the Tasmota command line informs us of the mqtt settings for this Tasmota device:

 MQT: stat/snifferbuddy/STATUS6 = {"StatusMQT":{"MqttHost":"gus","MqttPort":1883,"MqttClientMask":"DVES_%06X","MqttClient":"DVES_25EEA5","MqttUser":"DVES_USER","MqttCount":1,"MAX_PACKET_SIZE":1200,"KEEPALIVE":30,"SOCKET_TIMEOUT":4}}

The mqtt info lets us know the mqtt keep alive time is 30 seconds.

Install influxdb

InfluxDB (v1.8) is a time series-based database that is free to use on the Raspberry Pi. There can be multiple databases.

Gus as the database name

I stick with the database name of Gus. If you look in growbuddies_settings.json, you will see influxdb settings for the database.

sudo apt install influxdb-client Follow PiMyLifeUp’s directions to install.

Install Grafana

I followed the steps to install grafana. When I try:

sudo apt-get update

I got the message:

E: Conflicting values set for option Signed-By regarding source https://packages.grafana.com/oss/deb/ stable: /usr/share/keyrings/grafana-archive-keyrings.gpg !=
E: The list of sources could not be read.

after bumbling about on Google/StackOverflow, I ended up:

$ sudo nano /etc/apt/sources.list.d/grafana.list

and deleted the duplicate lines. Then sudo apt-get update worked.

Playtime

Note

You must build SnifferBuddy and Gus before playtime can begin.

If you have a SnifferBuddy that is sending out MQTT messages and a Gus device that is running an MQTT broker and InfluxDB database, you can access the command line of the Gus device and type the following command:

store-readings

This will start running the main function in the __main__.py file.

Storing SnifferBuddy Readings

Now that you have set up a SnifferBuddy device that is sending readings through MQTT messages, and you have an MQTT broker (Gus) in place to route those messages, it is time to store the data for post visualization.

This Python script utilizes the growbuddies.snifferbuddyreadings_code.SnifferBuddyReadings class to calculate the VPD and store SnifferBuddy readings in an InfluxDB database.

The script can be run from the command line (within your virtual environment) by typing store-readings.

This script performs the following tasks:

  1. Subscribes to mqtt messages from SnifferBuddy. See the main() method.

  2. When a SnifferBuddy message is received, it calculates the vpd (vapor pressure deficit).

  3. Stores the SnifferBuddy reading and the calculated vpd value in an influxdb database.

  4. Logs debug messages for easier debugging.

growbuddies.__main__.main()

The goal of this script is to store the SnifferBuddy readings along with the calculated vpd value in a measurement table within InfluxDB. To achieve this goal, the script defines two callback functions: one for handling SnifferBuddy readings and the other for handling status information.

The Settings class reads in parameters used by the GrowBuddies from the growbuddy_settings.json file. One of these parameters is a dictionary containing information for subscribing to SnifferBuddy’s MQTT topics. The MQTTService class uses this dictionary to determine which callback function to use when it receives a message. Specifically, it will use the readings callback if the message contains air readings, or the status callback if the message contains SnifferBuddy status information such as “online” or “offline”.

"snifferbuddy_mqtt":
                            {"tele/snifferbuddy/SENSOR": "on_snifferbuddy_readings",
                             "tele/snifferbuddy/LWT": "on_snifferbuddy_status"},

The above is the default entry. The topic is the dictionary’s key. The name of the callback function is the value. Notice this script includes the two methods Callbacks.on_snifferbuddy_readings() and Callbacks.on_snifferbuddy_status().

class growbuddies.__main__.Callbacks
__init__()
on_snifferbuddy_readings(mqtt_payload: str)

A message from SnifferBuddy is available via the MQTT protocol. The purpose of this callback is to store the reading in an InfluxDB database that has already been installed.

  1. Convert mqtt payload string into a SnifferBuddy() class.

A SnifferBuddy built with an SCD30 sensor will send out messages with payloads similar to:

{
    "Time": "2022-11-24T13:12:37",
    "ANALOG": {"A0": 1024},
    "SCD30": {
        "CarbonDioxide": 630,
        "eCO2": 661,
        "Temperature": 73.3,
        "Humidity": 50.0,
        "DewPoint": 53.5
    },
    "TempUnit": "F"
}

Before the readings are received by this callback, they are transformed into a sensor agnostic Python class, SnifferBuddyReadings().

s = SnifferBuddyReadings(tasmota_payload)
         print(f"Time: {s.time}, Temperature (F): {s.temp}, Humidity: {s.humidity})
         s.time = "2022-09-06T08:52:59"
         s.temperature = 71.8
         s.humidity = 61.6
         s.co2 = 814
         s.light_level = 542
         s.vpd = 1.23

The above example shows how easy it is to get individual values. Alternatively, s.dict returns a dictionary containing all of the values.

  1. Store the Readings into an influxdb measurement table.

Note

Gus must be running with the influxdb service installed in order to store readings.

The ReadingsStore class is used to store readings from SnifferBuddy. When initializing the ReadingsStore class, the following properties are read from the growbuddy_settings.json file:

self.hostname = self.settings.get("hostname")
self.db_name = self.settings.get("db_name")
self.table_name = self.settings.get("snifferbuddy_table_name")
on_snifferbuddy_status(status)

Note

How SnifferBuddy’s status is determined is discussed in Using MQTT To Determine Device Health.

“This callback function handles MQTT Last Will and Testament (LWT) messages regarding the online status of SnifferBuddy, which can be either “online” or “offline.”

Args:

status (str): Either the string “Online” or “Offline”.

Additional Python Modules

mqtt_code

class growbuddies.mqtt_code.MQTTService(callbacks_dict=None)

GrowBuddies code that subscribes to topics and runs as a systemd service will need to get to the MQTT client through MQTTService(). This is necessary because a systemd service requires a blocking call, such as loop_forever(), in order to run continuously in the background. However, the loop_forever() method prevents the thread from continuing. By using a separate thread for the MQTTClient, the service can run in the background without blocking the main thread.

Attributes:

callbacks_dict (dict): A dictionary mapping MQTT topics to callback functions. The callbacks_dict is created by calls to the growbuddies.settings_code.Settings class.

Methods:

start(): Starts the MQTT client in a separate thread.

stop(): Stops the MQTT client and waits for the thread to terminate.

Note

The default mqtt broker is Gus

__init__(callbacks_dict=None)
class growbuddies.mqtt_code.MQTTClient(callbacks_dict=None)

A more optimized way to publish and subscribe to the Gus MQTT broker.

If the calling code logic is subscribing to MQTT topics, then access to the MQTTClient() must go through MQTTService().

If the calling code logic is strictly publishing MQTT messages, then access to publishing mqtt messages can all be done through MQTTClient().

It also provides methods for handling incoming messages and performing cleanup when the client is stopped.

Attributes:

host (str): The hostname or IP address of the MQTT broker.

callbacks_dict (dict): A dictionary mapping MQTT topics to callback functions.

__init__(callbacks_dict=None)
on_message(client, userdata, msg) None

Callback function that is called when the client receives a message from the MQTT broker.

Args:

msg (str): An MQTT message.

To determine which callback function to call for a specific MQTT message topic, a callbacks_dict is used to map the topic to the appropriate callback function. The callbacks_dict is passed in by the caller and is created using the growbuddies.settings_code.Settings.get_callbacks() method. This allows for a generic way to handle different MQTT topics and their corresponding callback functions.

The callbacks_dict is searched to find the function that should be executed when the topic of the incoming MQTT message matches a key in the dict.

SnifferBuddyReadings

The snifferbuddy_readings.py module contains the SnifferBuddyReadings class.

class growbuddies.snifferbuddyreadings_code.SnifferBuddyReadings(mqtt_payload)

An instance of the “SnifferBuddyReadings” class simplifies access to data by processing the MQTT message payload from a SnifferBuddy device and providing properties such as the vpd, as well as a dictionary containing all properties. Using the raw MQTT payload may not be optimal, as it: - Does not include the vpd calculation. - Is specific to the sensor. - Is not as easily accessible as accessing properties directly.

For example, the mqtt air quality message of the SCD30 is:

{"Time":"2022-09-06T08:52:59",
 "ANALOG":{"A0":542},
 "SCD30":{"CarbonDioxide":814,"eCO2":787,"Temperature":71.8,"Humidity":61.6,"DewPoint":57.9},"TempUnit":"F"}

Another air quality sensor may have a different message format that requires interpretation. However, the process of accessing properties, such as sensor readings, remains consistent and can be done in the same way as demonstrated below.

from snifferbuddy_code import SnifferBuddyReadings
s = SnifferBuddyReadings(mqtt)
print({s.dict})
print({s.vpd})
Args:

mqtt_payload (str): Sensor model specific mqtt message from a SnifferBuddy. sensor (str, optional):The constant string that identifies the air quality sensor. The sensor must be listed in SnifferBuddySensors(). Defaults to SnifferBuddySensors.SCD30. log_level (logging level constant, optional):Logging level either logging.DEBUG, logging.INFO, logging.ERROR. Defaults to logging.DEBUG.

__init__(mqtt_payload)
property co2: float

SnifferBuddy’s reading for the CO2 concentration.

property dict: dict

Returns SnifferBuddy readings as a dictionary.

return {
    "temperature": self.temperature,
    "humidity": self.humidity,
    "co2": self.co2,
    "vpd": self.vpd,
    "light_level": self.light_level,
}

_Note: The time is not returned. influxdb adds a timestamp when the data is written to the database._

property humidity: float

SnifferBuddy’s reading for the Relative Humidity

property light_level: int

The reading of the photoresistor at the top of SnifferBuddy. The build directions say to use a Pull Down resistor. This means the lower the value of the light level, the higher the light level. The value is a number between 0 and 1023.

property temperature: float

SnifferBuddy’s temperature reading. Whether it is in F or C is dependent on how you set up SnifferBuddy. See [GrowBuddy’s Tasmota documentation](tasmota_commands) for details.

Returns:

float: SnifferBuddy’s reading of the air temperature.

property time: str

A string containing the date and time SnifferBuddy sent the MQTT message. _Note: The time is not returned within the dict property, as noted earlier.

property vpd: float

A calculation of the Vapor Pressure Deficit (vpd) based on SnifferBuddy’s temperature and humidity readings. _Note: For now, the leaf temperature is assumed to be 2 degrees F less than the air temperature._

Settings

The settings_code.py module efficiently retrieves and validates parameter input from the growbuddy_settings.json file.

class growbuddies.settings_code.Settings
__init__()

Set the working directory to where the Python files are located.

get(key, default=None) str

The get() method is a convenient way to access the values stored in the growbuddies_settings.json file. It serves as a wrapper around the built-in get() method of dictionaries. If a value does not exist for a key, but a default value is provided, the default value is returned. If a value does not exist for a key and no default value is provided, an exception is raised.

Args:

key (str): One of the keys in the growbuddies_settings.json file.

default (str, optional): A default value to return if the key does not exist. Defaults to None.

Raises:

Exception: Occurs if the key does not exist and no default value is provided. The exception message contains the key and the value.

Returns:

str: The value associated with the key.

get_callbacks(key: str, instance_of_callbacks_class) dict

get_callbacks() returns a dictionary of callback methods to be called by the growbuddies.mqtt_code.MQTTClient.on_message(). For example, the growbudies_settings.json file contains several callback dictionaries. Here is the callback dictionary for SnifferBuddy:

{
"snifferbuddy_mqtt": {
    "tele/snifferbuddy/SENSOR": "on_snifferbuddy_readings",
    "tele/snifferbuddy/LWT": "on_snifferbuddy_status"
    }
}

The dictionary states for the key “snifferBuddy_mqtt”, when a message is received on the topic “tele/snifferbuddy/SENSOR”, the method on_snifferbuddy_readings() is to be called. When a message is received on the topic “tele/snifferbuddy/LWT”, the method on_snifferbuddy_status() is to be called.

Typically, this method is called prior to instantiating an instance of the growbuddies.mqtt_code.MQTTService class. For example, the following code instantiates an instance of the MQTTService class and passes the callbacks dictionary to the MQTTService class:

settings = Settings()
settings.load()
callbacks = Callbacks()
methods = settings.get_callbacks("snifferbuddy_mqtt", callbacks)
mqtt_service = MQTTService(methods)
mqtt_service.start()
Args:

key (str): Key to the method names in the instance_to_callbacks_class instance that are to be called when a message is received on the topic associated with the key. In the above example, the caller has written the two callback methods, :on_snifferbuddy_readings() and on_snifferbuddy_status().

instance_to_callbacks_class (class instance): Instance of the class that contains the callback methods.

Returns:

dict (dict): A dictionary of callback methods to be called by growbuddies.mqtt_code.MQTTService. The dictionary is keyed by the topic and the value is the callback method to be called.

load() None

Load the growbuddies_settings.json file.

Raises:

Exception: Returns a string letting the caller know the json file could not be opened.

Useful Raspberry Pi Stuff

Installed Raspberry Pi But Cannot SSH

You’ve verified Gus has an IP address. However, perhaps you accidentally entered the wrong SSID or password for your wifi. Or you forget to enable SSH. You can manually configure these options.

  • Add “SSH” file to the root of the image. We do this by opening a terminal on the boot partition and typing $touch ssh

  • Create the wpa_supplicant.conf file : $touch wpa_supplicant.conf. Copy the contents into the file nano wpa_supplicant.conf:

country=US
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1

network={
    ssid="YOURSSID"
    psk="YOURPWD"
}

Note: Multiple wifi networks can be set up by following this example:

network={
    ssid="SchoolNetworkSSID"
    psk="passwordSchool"
    id_str="school"
}

network={
    ssid="HomeNetworkSSID"
    psk="passwordHome"
    id_str="home"
}

Changing the ssid and psk to match your network.

  • Remove the SD-card.

  • Put the SD-card into the Rasp-Pi’s micro-SD port.

  • Power up the Rasp Pi. Hopefully wireless is working!

Using Rsync

Rsync is a very useful utility on the Raspberry Pi. I document my use here because I keep forgetting how to use it. Currently, I am on a Windows PC. The challenge is to start a Bash session in the right directory.

  • open Explorer, go to the directory to use rsync, and type in bash in the text field for the file path.

_images/explorer_with_bash.jpg

Getting to Bash Command Line Through Explorer

A wsl window will open at this location.

sudo rsync -avh pi@gus:/home/pi/mydata.zip  .

Change Text

From within a directory within multiple files:

find /path -type f -exec sed -i 's/oldstr/newstr/g' {} \;

OSError: [Errno 98] Address already in use

Find the ProcessID

Using the command:

 pi@gus:~/gus $
pi        2465  0.2  0.4  25956 17520 pts/0    T    09:31   0:00 /home/pi/gus/py_env/bin/python /home/pi/gus/py_env/bin/sphinx-autobuild docs docs/_build/html
pi        2641  0.0  0.0   7344   508 pts/0    S+   09:34   0:00 grep --color=auto sphinx-autobuild

The PID we are interested in is 2465.

Kill the Process

Onto the kill command, which needs sudo privileges.

pi@gus:~/gus $ sudo kill -9 2465
[1]+  Killed                  sphinx-autobuild docs docs/_build/html  (wd: ~/gus/docs)
(wd now: ~/gus)

Check which Python is in Use

It’s best practice to run Python code in a virtual environment. To check to see which interpreter is running:

py_env) pi@gus:~/growbuddies $ python
Python 3.7.3 (default, Jan 22 2021, 20:04:44)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.executable
'/home/pi/growbuddies/py_env/bin/python'

Check Rasp Pi OS Version stuff

(py_env) pi@gus:~/growbuddies/growbuddies-project $ uname -a
Linux gus 5.10.103-v7l+ #1529 SMP Tue Mar 8 12:24:00 GMT 2022 armv7l GNU/Linux

(py_env) pi@gus:~/growbuddies/growbuddies-project $ cat /etc/debian_version
10.13

(py_env) pi@gus:~/growbuddies/growbuddies-project $ cat /etc/rpi-issue
Raspberry Pi reference 2021-12-02

Note

Debian 10 is Buster. Debian 11 is Bullseye.

Ignore a Page

If you don’t want to include a document in the toctree, add :orphan: to the top of your document to get rid of the warning.

This is a File-wide metadata option. Read more from the Sphinx documentation.

Packaging

There is so much history with Python. Resource: Python packaging.