# Problems occuring at high inject rates (\> 400 messages per second)
**URL:** https://discourse.nodered.org/t/problems-occuring-at-high-inject-rates-400-messages-per-second/44381
**Category:** Industrial
**Created:** [19 April 2021 06:48 UTC](https://discourse.nodered.org/t/problems-occuring-at-high-inject-rates-400-messages-per-second/44381 "2021-04-19T06:48:32Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@vapourlyric](https://discourse.nodered.org/u/vapourlyric)
#### Post date: [19 April 2021 06:48 UTC](https://discourse.nodered.org/t/problems-occuring-at-high-inject-rates-400-messages-per-second/44381/1 "2021-04-19T06:48:32Z")
High everyone!
I recently experienced a problem that only occurs when the inject rate is relatively high. It seems like messages are getting lost inside the flow.
I have an industrial hardware based on the Raspberry Pi 3 Compute Module. The Hardware has 2 CAN interfaces and is running Node-RED 1.2.9. I am utilizing red-contrib-socketcan to get the CAN messages from both CAN interfaces into Node-RED.
CAN messages often have a cyclic behaviour which leads to a lot of duplicates because messages are sent even if the payload has not changed since the last transmission. That is why I decided to implement a duplicate filter in a function node based on the JavaScript Map Object. I am aware that for this purpose the RBE node would be suitable. However, the RBE node has 2 disadvantages:
1. The RBE node relies on the msg.topic object. As far as I understood, the msg.topic object must be a string that is compatible with the MQTT topic namespace convention. However, I have not tried yet how the RBE node behaves if msg.topic is not a string but instead a javascript object.
2. The actual payload of a CAN message is of the type binary. In node.js, the corresponding data type is "Buffer". I believe that the RBE node is not able to compare 2 Buffer objects. However, I have not tried that yet.
In my function node I utilize context.get and context.set to get access to my Map Object. In a nutshell, first I read my Map Object with context.get, then I do the payload compare stuff, then I write my Map object with context.set and finally I decide if I pass the message with "return msg" or not, depending on whether I detected a duplicate. In other words: duplicate messages get dropped.
This is where it gets weird. Everything seems to work fine if I only have one hundred CAN messages per second or so. But when I increase the number of CAN messages on my CAN network to lets say 400 CAN messages per second, it seems that my function node occasionally also drops messages that are not duplicates. I reviewed my code but could not find any error.
So I decided to change my strategy. I utilized a join node to collect all incoming CAN messages in a 5 second time window. In a nutshell, the output of the join node is an array of CAN messages. That output is connected to a function node that loops through the array via a for loop. Inside the for loop I run the exact same code like before. The only difference is that the context.get (before the for loop) and context.set (after the for loop) operations are only done once for the whole batch. This strategy seems to work perfectly and I did not experience any non duplicates dropped even at 1000 messages per second. Obviously the disadvantage of this strategy are the 5 second batches which means that after the join node the message flow is not in realtime anymore.
My assumption is that Node-RED is **not** utilizing queues between two connected nodes and messages get overwritten if they have not been processed yet by the following node. I think this means also that the context.get and context.set methods would use relatively much CPU time.
In the blog post I found these articles that seem to be related to this issue:
> **[Knowing when a node is done](https://nodered.org/blog/2019/09/20/node-done)**
>
> There's a new callback function signature for handling messages in Node-RED 1.0. If you've created your own nodes, find out what's changing, what to update and how to stay backwards compatible.
> **[Making flows asynchronous by default](https://nodered.org/blog/2019/08/16/going-async)**
>
> Changes are coming in Node-RED 1.0 to how messages are routed in a flow. Find out what it means to go asynchronous.
However these articles did not confirm my assumption.
Can anybody clarify whether my assumption is true?
Is it possible that this issue is taken into account in future Node-RED version?
---
### Author:  [@dceejay](https://discourse.nodered.org/u/dceejay)
#### Post date: [19 April 2021 09:40 UTC](https://discourse.nodered.org/t/problems-occuring-at-high-inject-rates-400-messages-per-second/44381/2 "2021-04-19T09:40:48Z")
It could be possible, but shouldn't happen if you are only using the variables within the one function if nothing async is occurring in there. You could try the old way of using function context (that doesn't persists) which is just to add things directly to the context object - eg `context.foo = bar` rather than get and set.
Re your other questions - 1) the topics are just compared with == which won't work well with objects so I would stick with strings. and 2) buffers should compare just fine.
---
### Author:  [@vapourlyric](https://discourse.nodered.org/u/vapourlyric)
#### Post date: [19 April 2021 15:36 UTC](https://discourse.nodered.org/t/problems-occuring-at-high-inject-rates-400-messages-per-second/44381/3 "2021-04-19T15:36:03Z")
Thank you for the information. I did some tests with the rbe node. This is what I found out:
1. Using a javascript object for msg.topic does not work at all with the rbe node. However, when I convert the msg.topic javascript object with JSON.stringify, everything works as expected. In my case it looks like this: `msg.topic = JSON.stringify({iface:1,rtr:false,ext:false,id:0x181});` In this example the object is hard coded which is of course never the case in a productive scenario. It is of course also important that the keys are always the same and always in the same order. By utilizing the json node it is even possible to avoid custom code inside a function node. When I pass the message with the stringified msg.topic to an MQTT publish node, it seems to publish the message without complaining that it is a non valid MQTT topic. However, a much more useful MQTT topic for my example would be: "1/false/false/0x181" This could probably only be achieved by utilizing string manipulation methods within a function node. In the long term I would like to use the [Sparkplug B specification](https://cirrus-link.com/mqtt-sparkplug-tahu/) for MQTT topic namespace and payload. It would be great if Node-RED would support this out of the box (rather sooner than later). This would mean that Node-RED would be compatible with industrial software products like Ignition or Factory Studio. I guess I am a little off topic now 😏
2. The rbe node works absolutely fine with javascript buffers. I did not expect this because the == operator does not work on buffers as one would expect. Instead there is the Buffer.compare method as described [here](https://nodejs.org/api/buffer.html#buffer_static_method_buffer_compare_buf1_buf2). Maybe the rbe node uses exactly that method under the hood.
I will do some more tests to make sure that there are no non duplicate message drops in this scenario.
---
### Author:  [@krambriw](https://discourse.nodered.org/u/krambriw)
#### Post date: [19 April 2021 16:06 UTC](https://discourse.nodered.org/t/problems-occuring-at-high-inject-rates-400-messages-per-second/44381/4 "2021-04-19T16:06:30Z")
> [@vapourlyric](#):
>
> This would mean that Node-RED would be compatible with industrial software products
Is there anything, while using the available MQTT (and other) nodes, that actually prevents you from following the Sparkplug specification? I mean, MQTT does not specify any required topic namespace/structure or any particular payload data encoding within its implementation. So it could be it is possible to apply such "recipes" while configuring the nodes? But maybe to make it easier to target compatibility "out of the box", another set of IoT specificically dedicated MQTT nodes would be of interest. To simplify the setup, rapid engineering...
---
### Author:  [@vapourlyric](https://discourse.nodered.org/u/vapourlyric)
#### Post date: [19 April 2021 17:15 UTC](https://discourse.nodered.org/t/problems-occuring-at-high-inject-rates-400-messages-per-second/44381/5 "2021-04-19T17:15:14Z")
@krambriw Implementing the Sparkplug B specification is not an easy task. I began to study the details of this specification and came to the conclusion that this would be a pretty big project. The only library that is available (node-red-contrib-sparkplug) is very basic, not the latest version of the specification and has not been updated for more than 2 years. One could argue that the fact that there is no good implementation available for Node-RED, despite the acceptance for this specification in the industry, speaks for itself.
---
### Author:  [@krambriw](https://discourse.nodered.org/u/krambriw)
#### Post date: [19 April 2021 17:20 UTC](https://discourse.nodered.org/t/problems-occuring-at-high-inject-rates-400-messages-per-second/44381/6 "2021-04-19T17:20:44Z")
Node-RED is wide open for contributions. If you feel for it and have the competence to develop nodes..
---
### Author:  [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)
#### Post date: [19 April 2021 18:01 UTC](https://discourse.nodered.org/t/problems-occuring-at-high-inject-rates-400-messages-per-second/44381/7 "2021-04-19T18:01:13Z")
> [@vapourlyric](#):
>
> Implementing the Sparkplug B specification is not an easy task. I began to study the details of this specification and came to the conclusion that this would be a pretty big project
And there is the problem. Sparkplug probably mainly helps commercial industrial use and likely needs some resources to be made available to make it happen.
---
### Author:  [@dceejay](https://discourse.nodered.org/u/dceejay)
#### Post date: [19 April 2021 19:19 UTC](https://discourse.nodered.org/t/problems-occuring-at-high-inject-rates-400-messages-per-second/44381/8 "2021-04-19T19:19:42Z")
@vapourlyric - re topic - the next version of RBE will allow the "key" variable to be configured so you won't need to use `msg.topic` so could create a special property for filtering, that means you can leave topic for mqtt use as you want.
---
### Author:  [@vapourlyric](https://discourse.nodered.org/u/vapourlyric)
#### Post date: [20 April 2021 04:09 UTC](https://discourse.nodered.org/t/problems-occuring-at-high-inject-rates-400-messages-per-second/44381/9 "2021-04-20T04:09:00Z")
@dceejay That is great news. I am eager to try this as soon as the new rbe node is available.
---
### Author:  [@system](https://discourse.nodered.org/u/system)
#### Post date: [19 June 2021 04:09 UTC](https://discourse.nodered.org/t/problems-occuring-at-high-inject-rates-400-messages-per-second/44381/10 "2021-06-19T04:09:21Z")
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.