Azure IoT Edge – using Apache Nifi as a 3rd party container
Series
Part 1 - dotnet vs python vs node - temperature emission - who is cooler?
Part 2 - Developing modules
Part 3 - Custom Containers using Apache Nifi
Part 4 - Custom Module using TimescaleDB
Part 5 - Custom Module using Grafana
Intro
This is part 3 in a series starting here that runs through building an Azure IOT Edge solution. This post will run through setting up a 3rd party docker container for an edge deployment.
The code is located at: https://github.com/chrismckelt/edgy
In order to manage data flow & logic on the edge we will deploy a data orchestrator. There are quite a few choices on the market – for this demo we will use Apache Nifi https://nifi.apache.org/.
Apache Nifi describes itself as ‘An easy to use, powerful, and reliable system to process and distribute data.’
What will this module do?
The Nifi module will listen to the dot net, python & node module messages and when a temperature exceeds 25°C it will publish a message to activate the air conditioning. The over heating module will receive the message and turn the air conditioning on, thus decrease the temperature.
This work was inspired by the following project:
https://github.com/iotblackbelt/nifimodule
Setting up Nifi
To connect Nifi to the edge hub MQTT/AMQP broker (and not using the inbuilt SDKs with code), we need to authenticate with self signed certificates.
The easiest way to use self signed certificates for Azure IoT Edge is to follow this tutorial https://docs.microsoft.com/en-us/azure/iot-edge/tutorial-machine-learning-edge-05-configure-edge-device.
I use the docker image provided in that tutorial to generate my self signed certs. After following the tutorial I upload the ROOT CA to the Azure IoT Hub. After verification it should appear as below:
When working locally with the simulator you will use the generated certs from VS Code. These are found at C:\ProgramData\iotedgehubdev
Deployment files
For debugging purposes I made a deployment file that runs only the DotNetGenerator & Nifi. The sections below are also in the full local debug and production template.
Nifi config setup
When the Nifi docker file starts it will execute a script to do the following:
- copy existing files from the local computer using docker bind from c:/config to /config
- copy the self signed certificates to the Java cert store
- restart Nifi so the certs and new configuration apply
Nifi authentication
To connect and publish messages to the MQTT broker we add the following processors in our Nifi flow
- ConsumeMQTT
- PublishMQTT
Upon adding a processor you will need to enter the connection details and configure a SSL Context Service to connect to the edge hub broker.
SSL Context Service setting (certs used are stored in the Java cert store)
Settings
To generate a SAS token, right click on the device and select ‘Generate SAS Token for Device’
Nifi logic to air con control
Using the inbuilt IoT routing system temperature payloads published from dot net, python & node will be pushed to Nifi.
Nifi checks if the temperature recording exceeds 25°C. When this occurs it will publish its own message which is routed back to the overheating module requesting it to turn on the air con.
In Nifi this translates to the following processor flows (probably a better way to do this – Nifi experts?)
Nifi flows
Running the solution locally
Upon starting the solution each module will publish a temperature starting at 20°C . Nifi will receive each message via the Consume MQTT processor. Viewing the data provenance shows all received messages:
The modules keep publishing increasing temperatures until Nifi received a temperature over 25°C. Nifi then publishes a message requesting the respective module (e.g. PythonDataGenerator) to decrease the temperature (turn on the air con).
Watch out when publishing your container to ACR
Be careful when publishing your local containers to your remote container registry. I was publishing direct from my machine to my Azure Container Registry. Meanwhile the environment variables were set to look for certificates in the wrong place. So Nifi would not authenticate.
Outro
Here we have shown how to build and deploy a 3rd party container in our edge solution. Once we have the installed the custom certificates Nifi can authenticate to the edge hub and send/receive messages.
Published: