Stuck with change node

General
1

I'm having issues with the second change node. I can't seem to get the syntax right for it to work.

Its a simple flow, I may be abusing MQTT by embedding data in the topic, but it seems to work quite well except I can't create a message suitable for the file write node, here is the flow:

[{"id":"d4d6d7ff.db9e38","type":"mqtt in","z":"5abbd703.0270a8","name":"AIdetection","topic":"AIdetection/#","qos":"0","broker":"933ca107.c2a6","x":243,"y":739,"wires":[["6606d3f7.33608c","68d7b1fc.212cf","1ee26211.c8944e"]]},{"id":"68d7b1fc.212cf","type":"change","z":"5abbd703.0270a8","name":"","rules":[{"t":"move","p":"payload","pt":"msg","to":"image","tot":"msg"},{"t":"move","p":"topic","pt":"msg","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":472,"y":738,"wires":[["f1b2475e.179118","1ee26211.c8944e"]]},{"id":"f1b2475e.179118","type":"csv","z":"5abbd703.0270a8","name":"","sep":"!","hdrin":"","hdrout":"","multi":"mult","ret":"\\n","temp":"topic,folder,filename,startX,startY,endX,endY","skip":"0","x":677,"y":740,"wires":[["f4a30f0b.33bb4","1ee26211.c8944e"]]},{"id":"f4a30f0b.33bb4","type":"change","z":"5abbd703.0270a8","name":"","rules":[{"t":"set","p":"filename","pt":"msg","to":"payload.filename","tot":"msg"},{"t":"move","p":"image","pt":"msg","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":662,"y":854,"wires":[["4eb8acdc.d019c4","1ee26211.c8944e"]]},{"id":"4eb8acdc.d019c4","type":"file","z":"5abbd703.0270a8","name":"","filename":"","appendNewline":false,"createDir":true,"overwriteFile":"true","x":855,"y":846,"wires":[["c77ea48b.8c88d8"]]},{"id":"1ee26211.c8944e","type":"debug","z":"5abbd703.0270a8","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","x":864,"y":797,"wires":[]},{"id":"933ca107.c2a6","type":"mqtt-broker","z":"","name":"localhost","broker":"localhost","port":"1883","clientid":"","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""}]

This is the debug output from the MQTT input node, nothing special other than my data embedded in the topic:
MQTT message:
8/18/2019, 4:31:47 PMnode: 1ee26211.c8944e
AIdetection/!2019-08-18!16_31_47.2_Mendel_Cam2.jpg!409!234!492!421 :
msg : Object
object
topic: "AIdetection/!2019-08-18!16_31_47.2_Mendel_Cam2.jpg!409!234!492!421"
payload: buffer[362984]
qos: 0
retain: false
_topic: "AIdetection/!2019-08-18!16_31_47.2_Mendel_Cam2.jpg!409!234!492!421"
_msgid: "816e438e.c1ca"

Debug output from first Change node to move payload to image and topic to payload:
8/18/2019, 4:31:47 PMnode: 1ee26211.c8944e
msg : Object
object
qos: 0
retain: false
_topic: "AIdetection/!2019-08-18!16_31_47.2_Mendel_Cam2.jpg!409!234!492!421"
_msgid: "816e438e.c1ca"
image: buffer[362984]
payload: "AIdetection/!2019-08-18!16_31_47.2_Mendel_Cam2.jpg!409!234!492!421"

So far so good, I though I was home free with the output of csv node to split data embedded in MQTT topic:
8/18/2019, 4:31:47 PMnode: 1ee26211.c8944e
msg : Object
object
qos: 0
retain: false
_topic: "AIdetection/!2019-08-18!16_31_47.2_Mendel_Cam2.jpg!409!234!492!421"
_msgid: "816e438e.c1ca"
image: buffer[362984]
payload: array[1]
0: object
topic: "AIdetection/"
folder: "2019-08-18"
filename: "16_31_47.2_Mendel_Cam2.jpg"
startX: 409
startY: 234
endX: 492
endY: 421

Just use a second change node to move the msg.payload.filename to msg.filename move the msg.image back to msg.payload for input into the file write node.

Fail.
I feel pretty stupid, I've permuted what I could think of but just can't get it to work. Here is the output of the second Change node that I can't get correct:
8/18/2019, 4:31:47 PMnode: 1ee26211.c8944e
msg : Object
object
qos: 0
retain: false
_topic: "AIdetection/!2019-08-18!16_31_47.2_Mendel_Cam2.jpg!409!234!492!421"
_msgid: "816e438e.c1ca"
payload: buffer[362984]

msg.payload looks correct but msg.filename is nowhere to be found :frowning:

2

I was stupid.

Got it to work. I had the CSV node set to output "one message per row", an array. I don't understand the object array syntax so there is something more for me to learn, but changing the CSV to "one message per row" made it work.

Move msg.payload.filename to msg.filename made it work after the change to the CSV node. This change was my first try, it didn't occur to me the real issue was in the previous node.

I really like that the node-red file write creates the intermediate directories automatically, On the Python side this made for some ugly code and an if statement that was only true and executed once a day.

I'd still be grateful is someone could give me the correct syntax to access the payload array fields for future reference.

3

I may be abusing MQTT by embedding data in the topic, but it seems to work quite well

Possibly the understatement of the year. There is no reason to abuse the topic for topic-related data.
Just for fun, download mqttexplorer to see the mess.

Have a read about best practices.

2 Likes
4

Of course there is no need, but it was expedient to minimize the changes to the Python code to quickly get a data flow that I could use to concentrate and work on the node-red part of the system right now.

I'm a big believer in "rapid prototyping" to get something working quickly to identify the basic function and find potential issues.

Thanks for the mqttexplore link, I'd never heard of it before. It'll be fun and enlightening to poke around with it.

5

After looking at my "live" system with mqttexplorer I'm convinced that my quick little "hack" is close to perfect for my IOT system where everything is on a local, isolated subnet.

The behavior seems perfectly fine and I can change the "extra" data I transmit with one line of Python code to suit my needs for node-red side of things. If a "better" topic hierarchy simplifies the node-red side then and only then would more "invasive" Python side code changes be worthwhile.

Its not like I have to worry about overloading the broker, it exists only to do this task.

Despite being a master of understatement :slight_smile: I can't overstate the simple brilliance of MQTT and its amazing performance even on "weak" IOT hardware.

1 Like
6

Here’s one for future readers of this topic who found it through the search. Do not try this at home, the office or wherever else. There’s such a thing as best practice, and on the other end of the spectrum there’s a worst practice. I’m afraid this is one of those. This kind of needs would work well with a queue on Redis where you push to, and pop items from. For MQTT, the infrastructure is based on routing through the topic. If you use the topic to add metadata, you create unique routing endpoints instead with every single message published. MQTTExplorer will indeed show what a mess it will become, for lack of a kinder description.

A better way to do this would for example be the following, but gives a bit of additional overhead: take a checksum of the data you’re sending with a timestamp added, then publish those to a status topic with the metadata plus timestamp and checksum in the payload. If the data was not a buffer but a js object already you could add a key to store metadata and keep it in one message. Or slightly more work when parsing, prepend a binary fragment to your buffer in a standardised format with the meta inside, then split the data in meta and content upon receiving. That’s actually how a lot of protocols already work, from PDF to GIF files.

7

As long as you are not publishing retained messages and ideally only qos0 then the data would be ephemeral and this is a perfectly fine pattern. Not usual, but actually useful, as it does (as mentioned) make for easier parsing downstream,. Mqtt wildcard makes the subscription easy, and then you dont have mess with the payload.

1 Like
8

Yes they are QoS 0 messages, with retain False. It seemed patently obvious this was required for my hack to work.

To get back on topic about the change node, now that I've found my mistake, I'd like an example of using the change node with the payload.array. I couldn't seem to get it right, I'd like ot learn what I was doing wrong as it should also work and might be helpful in the future.

Another question, is it better or more efficient to use Set or Move to get the fields parsed out by the CSV node when I get them from the msg payload that ends up being replaced by the msg.image buffer at the end of the list with a Move. Or does it not matter?

9

Sorry I don't see anything close to resembling a mess of any kind, its been running for about 24 hours now.

mqttExplorer
mqttExplorer1030Ă—752 119 KB

10

For something as small as a file name it won’t really matter

11

There are four other numerical values as well as the filename. I don't have any instincts as to if one method is more efficient than the other. If the move is just changing a pointer to the data I don't see how it could matter but if it is copying and then deleting the old data I could see the Set being slightly more efficient as all the deletes would all happen at once at the end when the image is moved to the payload.

Just asking.

I'd still like an example of the syntax to move a field from the array object that my incorrect CSV node setting produced that messed me up initially.