Azure IoT Edge – using Apache Nifi as a 3rd party container
Part 1 - dotnet vs python vs node - temperature emission - who is cooler?
Part 2 - Developing modules
Part 3 - Custom Containers using Apache Nifi
Part 4 - Custom Module using TimescaleDB
Part 5 - Custom Module using Grafana
This is part 3 in a series starting here that runs through building an Azure IOT Edge solution. This post will run through setting up a 3rd party docker container for an edge deployment.
The code is located at: https://github.com/chrismckelt/edgy
Apache Nifi describes itself as ‘An easy to use, powerful, and reliable system to process and distribute data.’
The Nifi module will listen to the dot net, python & node module messages and when a temperature exceeds 25°C it will publish a message to activate the air conditioning. The over heating module will receive the message and turn the air conditioning on, thus decrease the temperature.
This work was inspired by the following project:
To connect Nifi to the edge hub MQTT/AMQP broker (and not using the inbuilt SDKs with code), we need to authenticate with self signed certificates.
The easiest way to use self signed certificates for Azure IoT Edge is to follow this tutorial https://docs.microsoft.com/en-us/azure/iot-edge/tutorial-machine-learning-edge-05-configure-edge-device.
I use the docker image provided in that tutorial to generate my self signed certs. After following the tutorial I upload the ROOT CA to the Azure IoT Hub. After verification it should appear as below:
When working locally with the simulator you will use the generated certs from VS Code. These are found at C:\ProgramData\iotedgehubdev
When the Nifi docker file starts it will execute a script to do the following:
- copy existing files from the local computer using docker bind from c:/config to /config
- copy the self signed certificates to the Java cert store
- restart Nifi so the certs and new configuration apply
To connect and publish messages to the MQTT broker we add the following processors in our Nifi flow
Upon adding a processor you will need to enter the connection details and configure a SSL Context Service to connect to the edge hub broker.
Using the inbuilt IoT routing system temperature payloads published from dot net, python & node will be pushed to Nifi.
Nifi checks if the temperature recording exceeds 25°C. When this occurs it will publish its own message which is routed back to the overheating module requesting it to turn on the air con.
In Nifi this translates to the following processor flows (probably a better way to do this – Nifi experts?)
Upon starting the solution each module will publish a temperature starting at 20°C . Nifi will receive each message via the Consume MQTT processor. Viewing the data provenance shows all received messages:
The modules keep publishing increasing temperatures until Nifi received a temperature over 25°C. Nifi then publishes a message requesting the respective module (e.g. PythonDataGenerator) to decrease the temperature (turn on the air con).
Be careful when publishing your local containers to your remote container registry. I was publishing direct from my machine to my Azure Container Registry. Meanwhile the environment variables were set to look for certificates in the wrong place. So Nifi would not authenticate.
Here we have shown how to build and deploy a 3rd party container in our edge solution. Once we have the installed the custom certificates Nifi can authenticate to the edge hub and send/receive messages.