Message structure convention

[bartoszek]

Hi!

I've got a problem with choosing a "Node-RED" way to "construct" the message passed through nodes.

To describe my "problem" I have an array of relays, couple of boards, with 16 relays each, most of them are used to control my floor heating, but there are also some that turn on the lights or does some other automations.
To do "low level" operations, I've created a subflow that accepts messages in form {relay:{id:42, operation:"on"}} and there are also "higher level" abstractions that accepts for example a mqtt message or something, to turn on the lights or heating, to keep things in order lets say that this it the 1st method.

But right now I'm thinking that information should be passed using the payload field, because 'this is the way' so the 2nd form would be {payload:{ relay:{id:42, operation:"on"}} } or just {payload:{id:42, operation:"on"}} (3rd) but... Now I'm thinking that we also have a topic, so it in the end, this might also be something like {payload:"on",topic:"/home/storey/0/kitchen/floor/heating"} (4th)

And now... I'm confused because if this is a Nore-RED way of sending messages, why there is no builtin mechanism of routing messages by topic like for example in MQTT? (a switch node which outputs the message on all fitting topics, not only one) Right now topic is only a field without any defined form, and from what I can see, most people threat this as a "text field" (a key) setCurrent, temperature, etc rather then a topic. (whereas as topic I understand the same as in mqtt topic)

1-st option, obsoletes the payload field, so I assume this is not OK
2-nd has a redundant field 'relay' which is a little bit pointless, but still acceptable for me
3-rd looks OK for me, but it has and ID field which, in my opinion, should be moved to topic field

all the above schemas can take additional field in payload/relay object (e.g 'delay', or 'momentary') without a type change, I just add another field and that's it

4-th looks more like messages that I've seen in other flows, but this has the limitation that adding new field changes the type of message and somewhere i need to check for datatype and have different logic to both flows.

5-th? where payload is not a text field containing an operation but rather all needed information? {payload:{operation:"on", something:0}, topic:"..."} ?

The question is... If you where in the same situation, which schema would you use, and WHY? what strategy should I use in the future?

There is a PEP for python, where is PEP 8 for Node RED?

Br,
Bartoszek

[E1cid]

If I understand correctly, Your scheme should make sense to you and be as simple as possible, the input message should contain all relevant info the id the topic and the payload. Then you just move the info to the correct location for the mqtt out node, or others.
e.g.

[{"id":"4d30d66c.154d88","type":"inject","z":"c74669a0.6a34f8","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"id\":1234,\"topic\":\"test/one\",\"value\":\"on\"}","payloadType":"json","x":80,"y":3980,"wires":[["e7111746.9208a"]]},{"id":"e7111746.9208a","type":"change","z":"c74669a0.6a34f8","name":"","rules":[{"t":"set","p":"topic","pt":"msg","to":"payload.topic","tot":"msg"},{"t":"set","p":"id","pt":"msg","to":"payload.id","tot":"msg"},{"t":"set","p":"payload","pt":"msg","to":"payload.value","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":260,"y":4020,"wires":[["31597a9a.887206"]]},{"id":"a26729ca.e35e18","type":"inject","z":"c74669a0.6a34f8","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"id\":5678,\"topic\":\"test/two\",\"value\":\"on\"}","payloadType":"json","x":70,"y":4060,"wires":[["e7111746.9208a"]]},{"id":"6ca36ddb.c9d1e4","type":"inject","z":"c74669a0.6a34f8","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"id\":1234,\"topic\":\"test/one\",\"value\":\"off\"}","payloadType":"json","x":70,"y":4020,"wires":[["e7111746.9208a"]]},{"id":"31597a9a.887206","type":"debug","z":"c74669a0.6a34f8","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":500,"y":4020,"wires":[]}]

In the above example I feed in payload with id, mqtt topic and a payload value.
The change node moves all items to correct location and then moves payload value to the payload. so the payload JSON would be

{"id":1234,"topic":"test/one","value":"on"}

The routing is done in the mqtt broker.

[Steve-Mcl]

@bartoszek

I'm confused because if this is a Nore-RED way of sending messages, why there is no builtin mechanism of routing messages by topic like for example in MQTT? (a switch node which outputs the message on all fitting topics, not only one)

Filtering like MQTT is a decent idea - but don't forget the switch node has all these ways of filtering

and on top of that you can use JSONata e.g. ($substring(topic, 1, 22) = "home/storey/0/kitchen/" to filter

@bartoszek

1-st option, obsoletes the payload field, so I assume this is not OK

It is perfectly fine to do this & sometimes preferable if the next node overwrites payload and you want to keep the original value in the msg for downstream operations. In truth, you will end up using topic and payload in many circumstances because many nodes don't employ the typedInput (an input that lets you specify where the input data is coming from).

@bartoszek

The question is... If you where in the same situation, which schema would you use, and WHY? what strategy should I use in the future?

It really depends on what you are doing. I say, dont bend to any convention that makes it less readable or maintainable. Personally, I add "meta" objects to my messages at the start of a flow (and on route) so that when it reaches the end of the flow I have a means of understanding what happened to the msg on its travels through the flow. This really helps with debugging.

The bottom line is, you are not constrained. Use the topic and payload methodology if it suits your application, otherwise do what is best for you.

[bartoszek]

Hi!
Thank you both for examples

@E1cid sadly this is not what I had hope for I know that there is a MQTT support in the node red itself, but the broker is not a first class citizen of node red (is not builtin) and also... I do not want to use MQTT in all "trafic" in NR itself, this is why we have flows and wires to not use external tools, right?

@Steve-Mcl I know that there are some mechanisms in switch node itself to check the structure of a topic, but still they are not so straightforward as simple #+ wildcards in MQTT that is why I'm confused, because there is no reason not to include them IF the use-case is similar to MQTT's topics...

And @Steve-Mcl you'r answer is, for me, the worst I can think of because if I should use what I need (or what fits me best) there is no reason whatsoever of having payload and topic field in the first place... It looks like someone had an idea and lost interest in it half way through But the end note that you added is great, I have an idea on how to handle my messages, similar to yours.
But still this is a little bit odd for me that there are no rules like for example
"use payload to store your data, topic to store the location", "topic should have a structure as ...", "copy content of payload to msg._orig in every subflow to properly handle errors" "use topic config/... to pass config to nodes" "don't do xxx because you will burn yourself' or something

still, thank you for answers

Br,
Bartoszek

[Steve-Mcl]

@bartoszek

And @Steve-Mcl you'r answer is, for me, the worst I can think of because if I should use what I need (or what fits me best) there is no reason whatsoever of having payload and topic field in the first place

It makes no difference. As I said before, use topic+payload or dont. The fact you dont does not make it the "worst". It is what it is. If you bend your application to suit the topic+payload at the cost of readability or maintainability then you will lose out. If you feel sticking to topic+payload is better for you and other maintainers, then do so.

@bartoszek

It looks like someone had an idea and lost interest in it half way through

I cant speak for the designers but I suspect that since node-red was/is heavily based on IoT (and thus MQTT) topic and payload were chosen as sensible means of identity+data. Then as more features were added, the convention was carried through.

The fact is there are no hard and fast rules. This is, in some ways a strength. Others coming from strict / typed languages will find this (much like JavaScript) annoying at first. I did too. But you learn to live with and eventually end up wishing those strongly types languages were as flexible as JavaScript

[Urs-Eppenberger]

Thanks for bringing this discussion up, I find this interesting.

I use MQTT for the communication between node-red and the sensors and actors. It is used too to for communication between flows, but less and less often.

I specified a naming convention for the data and use this for the topic tree in MQTT too. Inside node-red I do a mapping from the sensor data tree to a place-based data tree. Via the command data tree I send out commands to the actors:

sensor/'device-name'/'sensor-type' = 'value' sensor/sonoffth1/temp = 20.0 or sensor/sonoffpow1/relay1 = "off"
data/'place'/'measurement-type'='value' data/kitchen/temp = 20.0 or data/cellar/light = "on"
command/'device-name' = 'command'

There are other conventions, for example the homie convention which might be useful in your application.

My point is, do not just think about the message format in the flows of node-red, but consider the communication with your devices too, and use the same message structure.

[Christian-Me]

@Urs-Eppenberger great you mentioned the homie convention. Im a big fan of the homie convention - It gives a clear structure including mechanism for setting and reporting, auto device detection, status reporting and so on. Together with retained messages the mqtt broker is my state storage and reporting system. (I have around 20 remote devices (ESP8266 and ESP32) and 2-3 Node-RED instances perfectly in sync) I transfer Homematic, Zigbee, Modbus and other systems to homie first - in order that my logic flows are independent of the systems currently used or used in the future.

[bartoszek]

Hi.
Thank you for the work on homie for NR! I started to use it and I think this might suite me well enough.

That said I have a question (not sure if I should ask it here as it is not really related to thread topic anymore... ) why the "device" section of topic is linked with MQTT configuration? I have multiple devices connected to my RPI, and according to the spec. I should be able to "connect" them to different devices in the homie topic structure. But the topic root is in mqtt config so... I need to add multiple configurations of THE SAME mqtt server to achieve this, is this the desired way? or I just misunderstand the general idea?

[Christian-Me]

If you just started with the homie-convention node perhaps you like to start with the complete reworked version.

This supports beside many new features support for multiple root topics with a single mqtt config.

If you like you can download the dev version form github by

npm i Christian-Me/node-red-contrib-homie-convention#develop

homie is defined as root/device/node/property In the npm version you have to define a config for every different root topic. The dev version detects homie root topics by the mechanism described (somewhere) in the convention.

I already use the new version. As it will be very likely a breaking change I plan to release the new version only after a reasonable test phase. If you need more support I'm more than happy to help (all beta testers welcome). Either on GitHub or as a new topic here in the forum.