MQTT and Environment Variables

[ThisEndUp]

Hi.
I can save you reading the long screed below by distilling the current problem to this:
I edit the mqtt-broker node's Service parameter to read {MQTT_HOST}, which has successfully been defined to be alpha.beta.com. The attempt to connect to the broker fails, because it tries to connect to Whoever@mqtt://{MQTT_HOST}:1883
TL;DR details below. Help gratefully accepted.
~R

I looked in /lib/systemd/system/node-red.service and saw
ExecStart=/usr/bin/node-red.sh
so I added a line to /usr/bin/node-red.sh, and it now reads

#!/bin/bash -
source /home/me/bbbw.sh
export NODE_PATH=/usr/local/lib/node_modules
cd /usr/local/lib/node_modules/node-red/
/usr/bin/nodejs --max-old-space-size=128 red.js

where /home/me/bbbw.sh has
export MQTT_BROKER=alpha.beta.com
export MQTT_PORT=1883
export MQTT_CLIENT_ID=Whoever
export MQTT_TOPIC=Whatever
export MQTT_NAME=Machine

config-pin p9.11 uart
config-pin p9.13 uart
stty -F /dev/ttyO4 sane speed 2400

I have also modified settings.js to start with:
console.log("==============================================")
console.log("Starting node-red.")
console.log("broker: " + process.env.MQTT_BROKER);
console.log("client id: " + process.env.MQTT_CLIENT_ID);
console.log("topic: " + process.env.MQTT_TOPIC);
console.log("=+=-=+=-=+=-=+=-=+=-=+=-=+=-=+=-=+=-=+=-=+=-=+=-=+=-")

Now for the problems. When I halt and restart my beaglebone, which automatically starts node-red, the only part of the console log that I added that shows up is the last line (=+=-...), nor does the broker connection succeed.

So I stop and then start node-red manually. Running journalctl -f -u node-red -o cat
shows the result of the first couple of lines of settings.js, indicating that the environment variables have been expanded. Other tests show that the pin configuration has worked. In other words, the
source /home/me/bbbw.sh
line seems to have worked just as desired.

That's nice, but even after that the connection to the broker fails. The client id works, but not the host.

I am aware of the limitation "This only works if it replaces the entire property", but the mqtt broker node property in question is Server, and I provide ${MQTT_HOST}. Something else prepends this with mqtt://, perhaps making it look like I am not specifying the entire property to whatever is processing it.

Here's what I get for a manual restart [apologies for the length...]

Started Start Node-RED.
2400
==============================================
Starting node-red.
broker: alpha.beta.com
client id: Whoever
topic: Whatever
=+=-=+=-=+=-=+=-=+=-=+=-=+=-=+=-=+=-=+=-=+=-=+=-=+=-
1 Mar 14:09:39 - [info]
Welcome to Node-RED
===================
1 Mar 14:09:39 - [info] Node-RED version: v0.19.5
1 Mar 14:09:39 - [info] Node.js  version: v6.16.0
1 Mar 14:09:39 - [info] Linux 4.14.67-ti-r73 arm LE
1 Mar 14:09:43 - [info] Loading palette nodes
1 Mar 14:09:46 - [warn] rpi-gpio : Raspberry Pi specific node set inactive
1 Mar 14:09:46 - [warn] rpi-gpio : Cannot find Pi RPi.GPIO python library
1 Mar 14:10:02 - [info] Dashboard version 2.13.0 started at /ui
1 Mar 14:10:03 - [info] Settings file  : /root/.node-red/settings.js
1 Mar 14:10:03 - [info] Context store  : 'default' [module=memory]
1 Mar 14:10:03 - [info] User directory : /root/.node-red
1 Mar 14:10:03 - [warn] Projects disabled : editorTheme.projects.enabled=false
1 Mar 14:10:03 - [info] Flows file     : /root/.node-red/flows_beaglebone.json
1 Mar 14:10:03 - [info] Server now running at http://127.0.0.1:systemd/
1 Mar 14:10:03 - [warn]
---------------------------------------------------------------------
Your flow credentials file is encrypted using a system-generated key.
If the system-generated key is lost for any reason, your credentials
file will not be recoverable, you will have to delete it and re-enter
your credentials.
You should set your own key using the 'credentialSecret' option in
your settings file. Node-RED will then re-encrypt your credentials
file using your chosen key the next time you deploy a change.
---------------------------------------------------------------------
1 Mar 14:10:04 - [info] Starting flows
1 Mar 14:10:05 - [info] [wifiscan:e07da6ad.5061d8] wifiscan - Initialized
1 Mar 14:10:05 - [info] Started flows
1 Mar 14:10:05 - [info] serial port /dev/ttyO4 opened at 2400 baud 8N1
1 Mar 14:10:05 - [info] [mqtt-broker:BBBW] Connection failed to broker: Whoever@mqtt://${MQTT_HOST}:1883
1 Mar 14:10:20 - [info] [mqtt-broker:BBBW] Connection failed to broker: Whoever@mqtt://${MQTT_HOST}:1883
1 Mar 14:10:36 - [info] [mqtt-broker:BBBW] Connection failed to broker: Whoever@mqtt://${MQTT_HOST}:1883

[knolleary]

You have set MQTT_BROKER but you are trying to use MQTT_HOST in the node's configuration.

[ThisEndUp]

Thank you, thank you, thank you. I did learn a lot about how to set up the parameters, which I posted in the hope others will find that useful, but it took a second set of eyes to spot the glaring error.
Thanks!
Richard

[ThisEndUp]

Help! My "cure" is now broken.

In trying to use Tim Falken's sense-rsa package, I encountered the "TypeError: generateKeyPair is not a function" error. Googling convinced me that upgrading Node-Red would fix this. So, I followed the directions at https://nodered.org/docs/getting-started/beaglebone to update Node-Red. I'm now successfully running version 1.0.4.

However, my prior "cure" relied on systemd starting node-red via
/lib/systemd/system/node-red.service, which had the line
ExecStart=/usr/bin/node-red.sh
So I modified /usr/bin/node-red.sh

Well, this no longer works!
/lib/systemd/system/node-red.service has become /lib/systemd/system/nodered.service
and it uses
ExecStart=/usr/bin/env /opt/node-red/bin/node $NODE_OPTIONS /opt/node-red/lib/node_modules/node-red/red.js $NODE_RED_OPTIONS

[Aside: Having never heard of /usr/bin/env, I was mystified by what this was supposed to do, but apparently it runs /opt/node-red/bin/node with the arguments on the rest of the line.]

This confuses me, because node is neither nodered nor node-red. In fact, neither of these exists as a command anymore. And node is not a shell file, so I can't add the
source /home/me/bbbw.sh
line that I used before. As you can see from my prior comments, sourcing bbbw.sh does more than set environment variables: it also configures serial ports, so I can't see how to replace it. My attempt to use
ExecStartPre=source /home/me/bbbwgb.sh
did no good.

Can anyone help me update my old cure to the new system?

Thanks,
Richard

[dceejay]

The Beaglebones have their own way of installing and upgrading as per the docs... and you are now in that mode...

You should be able to change that to point back to your node-red.sh file - as long as that then calls the /usr/bin/env line to start Node-RED - with the $NODE_OPTIONS and $NODE_RED_OPTIONS set as you wish.

[ThisEndUp]

Thanks. It took a couple of tries, but ultimately I got the new version working again. The primary things that made life difficult, even afterI found nodered.service, were:

1. it was necessary to comment out the User= and Group= lines of the service file so that the nodered.sh file I created ran with root privileges, allowing it to configure the pins from GPIO to UART.
2. settings.js is not at ~root/.node-red/ but rather at /var/lib/node-red/.node-red

If there's someplace where these changes are explained, I didn't find it.

If there is a reasonable place to put this information, I suggest these three additions: (i) the change in name and location of nodered.service, (ii) the fact that it runs as user node-red and doesn't have root privileges unless nodered.service is modified, and (iii) the new location of settings.js.

Many thanks for all your help! I couldn't have managed this without it.
Richard