Using MQTT for publishing environmental data

In this article I show how I started using mosquitto on a Raspberry Pi and an Ubuntu VM.

The basics

One of the easiest way to start using MQTT is using a public MQTT broker. I chose to use test.mosquitto.org. A minimal but fun scenario to use MQTT on a debian/Ubuntu based Linux system involves:

  • Install MQTT command line tools:
    sudo apt-get install mosquitto-clients
  • Point your browser at a MQTT consumer widget at mosquitto.org:
    http://test.mosquitto.org/gauge/
  • Publish some data to the temp/random topic at the MQTT broker at mosquitto.org:
    mosquitto_pub -h test.mosquitto.org -t temp/random -m 23.0

And see how the gauge changes to the value you just submitted.

You can also start your own subscriber for the temp/random topic at  test.mosquitto.org:
mosquitto_sub -h test.mosquitto.org -t temp/random
You will see the value that was last sent to the topic, which is probably the value you just sent (but someone else might have sent another value in the mean time which should then also be reflected at the gauge you still have open in your browser). It seems the temp/random topic will always return 11. as the first value, after that you will receive the values you (or someone else) published on the subject. Just publish a few different values using the mosquitto_pub client program and see how both subscribers report the new values.

This shows the basic workings of a message queuing solution like the MQTT based Mosquitto implementation. The basic components are:

  • One or more publishers publishing data ‘topics’ and sending them to a broker.
  • One or more subscribers subscribing at the broker for certain events on a topic (like arrival of published data for some topic).
  • The broker receiving data and events for certain topics from publishers and distributing that data to subscribers for that data or events.

In the case above we have a publisher (the mosquitto_pub client program) publishing data for the topic temp/random to a broker at test.mosquitto.org and 2 subscibers for that data (the gauge and the mosquitto_sub client program).

Your own Broker

Let’s set up our own broker and include python Mosquitto bindings so we can write some scripts:

sudo apt-get install mosquitto python-mosquitto

Now we can again use mosquitto_sub and mosquitto_pub publishing our familiar topic on our own server and subscribing to changes, use separate shell sessions to subscribe a client and use another client to publish:
mosquitto_sub -h localhost -t temp/random
mosquitto_pub -h localhost -t temp/random -m 21.0

You probably want to set up logging for the mosquitto daemon (which is probably disabled by default), In  /etc/mosquitto/mosquitto.conf adjust the log_dest and log_type entry like:
log_dest syslog
log_type error
log_type warning

Or any other valid entries like documented in the configuration file itself. If you want to see on which ports and addresses the daemon listens then you also need log_type information. This can be handy when troubleshooting certain issues.

Your own temperature Gauge

Only using the command line clients is quite boring off course, so you might want to host your own temperature gauge. Assuming you have a apache (or other webserver) already running on your own Linux host you can take the following steps:

  • Change your working directory to the Documentroot of your webserver, e.g:
    cd /var/www
  • Clone the github repository for the temperature gauge:
    git clone https://github.com/jpmens/tempgauge.git
  • Adjust the hostname parameter in the tempgauge/index.html file from “localhost” to the hostname or IP address of your MQTT Mosquitto server host. e.g change
    var client = new Messaging.Client("localhost", 18883, to
    var client = new Messaging.Client("10.1.1.123", 18883,

As the temperature gauge uses Websockets for communication with the MQTT server, websockets should be enabled in the MQTT server. For Mosquitto you need to add the following lines to the  /etc/mosquitto/mosquitto.conf file:

listener 18883
protocol websockets

It might be that your Linux distribution still includes an old version of Mosquitto that doesn’t yet include websockets, if that is the case have a look at Building Mosquitto 1.4 which explains in detail how to get websockets enabled in Mosquitto. (note that this method might conflict with the Mosquitto server that was installed by your package manager, please remove it first and beware of the location of your new mosquitto server). You now might have to start Mosquitto by hand instead of via the init.d or service scripts.

Now load the temperature gauge in your browser (e.g. use http://10.1.1.123/tempgauge) and publish a new value to the MQTT server:  mosquitto_pub -h 10.1.1.123 -t temp/random -m 18. If all went well you will see the gauge responding to your published values for the /temp/random subject on your Mosquitto server.

Related Stuff

Off course there is much more to using MQTT than what can be touched on in a small blog article. I will try to devote another article to related subjects like:

  • Security
  • QoS
  • Topic naming conventions
    • e.g. Physical versus Semantical naming schemes

References

Terminology

Note that the terminology used in this article is not always according to MQTT terminology: e.g. the terms ‘data’ or  ‘value’ should probably be replaced by ‘publication’.

 

Advertenties

Using emoncms for monitoring environmental parameters

For some time I have been monitoring environmental parameters like temperature and humidity using cheap (but quite precice) sensors from weatherstations of Auriol and Oregon Scientific. For this I had to write some sensor callbacks in the great rtl_433 SDR application (see the github page for rtl_433). I will write an article on that later.

The measurements from these sensors are currently simply written to files including a timestamp. A perl script continuously processes the output sent to these files (using a simple tail -f unix pipe construct). Currently I’m storing the data in the Graphite/Whisper database. This software also has a front-end for displaying the data, it is however quite rudimentary and generates quite simple graphs.

I started using Orion which can use the Graphite interface as a backend and the incredible highcharts javascript library for displaying graphs.

Although the graphs look really nice and flashy they fall short if you want to have a quick overview of the current status. While looking for some nice widgets for displaying the instantaneous data I stumbled accross emoncms. Besides the nice widgets I like the way it is set up and the interfaces it offers. I installed it using the github approach.

It was easy to adapt my perl scripts which were publishing data to the graphite/whisper backend.To also publish to my locally installed emoncms server using its API. Some examples of available visualisations can be found on github.

Now that I was publishing the data to multiple backends I started thinking about better ways to do this. When adding a new sensor I was modifying the script that was also publishing the data to the backends to understand the new sensor output (rtl_433 currently doesn’t publish sensor data in a standardized way), I added a new routine that published the extracted data to the new emoncms backend. This seemed like something for which the Observer (Publish/Subscribe) pattern would be a good fit (see also  “Design Patterns” or “Head First Design Patterns“).

Several message queuing solutions exist for this, looking around for suitable alternatives it seemed like MQTT would be a good fit for this. The most critical part in an MQTT implementation is the Broker. Looking at open source implementations it seemed like Mosquitto would be a good choice because it seemed most suitable for the Raspberry Pi that I was already using as an rtl_433 SDR. A nice writeup of its installation can be found at the site of Jan-Piet Mens.

Todo;