I recently experienced a problem that only occurs when the inject rate is relatively high. It seems like messages are getting lost inside the flow.
I have an industrial hardware based on the Raspberry Pi 3 Compute Module. The Hardware has 2 CAN interfaces and is running Node-RED 1.2.9. I am utilizing red-contrib-socketcan to get the CAN messages from both CAN interfaces into Node-RED.
CAN messages often have a cyclic behaviour which leads to a lot of duplicates because messages are sent even if the payload has not changed since the last transmission. That is why I decided to implement a duplicate filter in a function node based on the JavaScript Map Object. I am aware that for this purpose the RBE node would be suitable. However, the RBE node has 2 disadvantages:
The RBE node relies on the msg.topic object. As far as I understood, the msg.topic object must be a string that is compatible with the MQTT topic namespace convention. However, I have not tried yet how the RBE node behaves if msg.topic is not a string but instead a javascript object.
The actual payload of a CAN message is of the type binary. In node.js, the corresponding data type is "Buffer". I believe that the RBE node is not able to compare 2 Buffer objects. However, I have not tried that yet.
In my function node I utilize context.get and context.set to get access to my Map Object. In a nutshell, first I read my Map Object with context.get, then I do the payload compare stuff, then I write my Map object with context.set and finally I decide if I pass the message with "return msg" or not, depending on whether I detected a duplicate. In other words: duplicate messages get dropped.
This is where it gets weird. Everything seems to work fine if I only have one hundred CAN messages per second or so. But when I increase the number of CAN messages on my CAN network to lets say 400 CAN messages per second, it seems that my function node occasionally also drops messages that are not duplicates. I reviewed my code but could not find any error.
So I decided to change my strategy. I utilized a join node to collect all incoming CAN messages in a 5 second time window. In a nutshell, the output of the join node is an array of CAN messages. That output is connected to a function node that loops through the array via a for loop. Inside the for loop I run the exact same code like before. The only difference is that the context.get (before the for loop) and context.set (after the for loop) operations are only done once for the whole batch. This strategy seems to work perfectly and I did not experience any non duplicates dropped even at 1000 messages per second. Obviously the disadvantage of this strategy are the 5 second batches which means that after the join node the message flow is not in realtime anymore.
My assumption is that Node-RED is not utilizing queues between two connected nodes and messages get overwritten if they have not been processed yet by the following node. I think this means also that the context.get and context.set methods would use relatively much CPU time.
In the blog post I found these articles that seem to be related to this issue:
However these articles did not confirm my assumption.
Can anybody clarify whether my assumption is true?
Is it possible that this issue is taken into account in future Node-RED version?
It could be possible, but shouldn't happen if you are only using the variables within the one function if nothing async is occurring in there. You could try the old way of using function context (that doesn't persists) which is just to add things directly to the context object - eg context.foo = bar rather than get and set.
Re your other questions - 1) the topics are just compared with == which won't work well with objects so I would stick with strings. and 2) buffers should compare just fine.
Thank you for the information. I did some tests with the rbe node. This is what I found out:
Using a javascript object for msg.topic does not work at all with the rbe node. However, when I convert the msg.topic javascript object with JSON.stringify, everything works as expected. In my case it looks like this: msg.topic = JSON.stringify({iface:1,rtr:false,ext:false,id:0x181}); In this example the object is hard coded which is of course never the case in a productive scenario. It is of course also important that the keys are always the same and always in the same order. By utilizing the json node it is even possible to avoid custom code inside a function node. When I pass the message with the stringified msg.topic to an MQTT publish node, it seems to publish the message without complaining that it is a non valid MQTT topic. However, a much more useful MQTT topic for my example would be: "1/false/false/0x181" This could probably only be achieved by utilizing string manipulation methods within a function node. In the long term I would like to use the Sparkplug B specification for MQTT topic namespace and payload. It would be great if Node-RED would support this out of the box (rather sooner than later). This would mean that Node-RED would be compatible with industrial software products like Ignition or Factory Studio. I guess I am a little off topic now
The rbe node works absolutely fine with javascript buffers. I did not expect this because the == operator does not work on buffers as one would expect. Instead there is the Buffer.compare method as described here. Maybe the rbe node uses exactly that method under the hood.
I will do some more tests to make sure that there are no non duplicate message drops in this scenario.
Is there anything, while using the available MQTT (and other) nodes, that actually prevents you from following the Sparkplug specification? I mean, MQTT does not specify any required topic namespace/structure or any particular payload data encoding within its implementation. So it could be it is possible to apply such "recipes" while configuring the nodes? But maybe to make it easier to target compatibility "out of the box", another set of IoT specificically dedicated MQTT nodes would be of interest. To simplify the setup, rapid engineering...
@krambriw Implementing the Sparkplug B specification is not an easy task. I began to study the details of this specification and came to the conclusion that this would be a pretty big project. The only library that is available (node-red-contrib-sparkplug) is very basic, not the latest version of the specification and has not been updated for more than 2 years. One could argue that the fact that there is no good implementation available for Node-RED, despite the acceptance for this specification in the industry, speaks for itself.
And there is the problem. Sparkplug probably mainly helps commercial industrial use and likely needs some resources to be made available to make it happen.
@vapourlyric - re topic - the next version of RBE will allow the "key" variable to be configured so you won't need to use msg.topic so could create a special property for filtering, that means you can leave topic for mqtt use as you want.
