I'm testing a NodeRed (1.2.7) Flow with data from an acceleration sensor with 2000Hz. The custom sensor-node has several outputs, which deliver data for acc_x, acc_y, acc_z.
The sensor sends data packages of 16 values per axis every 8ms. The packages get unpacked and send individually over the node outputs.
So something like:
send first value
send second value
send third value
.
.
.
send sixteenth value
--short break--
next 16 values
it seems the nodes after the sensor node get overrun in a strange way. For the output that sends data first over the first registered connection to a next node, the next node (for example a file node to check the values) receives 16 times the 16th value. But if the same output is also connected to another node, this node receives values 1-16...
To explain:
2khztest1 displays 16 times value 16
2khztest2 displays values 1-16
2khztest3 displays values 1-16 for the 3 joined messages from acc_x,y and z
it also seems to make a difference which node i connect first to the output. so if i connect accelerationx first, 2khztest2 will display 16 times values 16, if i connect 2khztest1 first it will behave as explained above.
I believe it isn't an issue with the custom node because NodeRed is able to get the correct values of the same output.
Did something like this occur to others? It just happens at high frequency, same flow with 10 Hz has worked fine. Are 2000Hz too much for NodeRed?
It is a little bit difficult to understand what your problem is without seeing input and output data and your flow but if I was to guess it is how you've configured the join node.
If you can share your flow and some sample data, we could have a look for you.
As for processing at 2000 khz and writing each one to file, I suspect this is pushing the limits. If you were to buffer in memory then flush to file less frequently, you might achieve this. I might be wrong but I'm pretty sure the file node opens and closes the file on each operation.
What is your end goal, to store all of this data to file? Send all of this data to a database? Or is it your intention to analyse the data in real-time to a summary data value?
Actually the problem probably is with the sensor node... By default node.js passes by reference so if they re-use the same msg object for each value they send then it will be overwritten by any later arriving value. When you fork the flow (by wiring two wires) then Node-RED automatically clones the message for the second wire so then it won't get changed... and so yes which is first and which is second does depend on the order in which you wire them up... to avoid this the sending sensor node really needs to clone the message before sending anything.
Only if you haven't fixed the filename - If it knows it's the same file name (because you have set it) - it should leave it open between writes.
Thanks for your answers @dceejay and @Steve-Mcl ! My end goal is to process the data before sending it to a database. Because it seems InfluxDB doesn't work at 2kHz and I want to reduce the data volume before moving it to a database. But for now, I first need to make sure I receive correct data at the start of the flow. step by step
I used the file node to have a better overview of the values as an alternative to the debug window. But nice to know that the file is left open!
The problem I described occurs before the message enters the join node, i also saw the same behavior when connecting the file node directly to the output.
if I add any node to also send the data to it (like before a file node or now a switch node with no output) and connect it to the output before i connect the change node of acc_x the file output displays 16 different values for x...
well, since you are "in the code" you could fix it and raise a PR?
PS, look at where the author calls node.send - you might want to do a RED.util.cloneMessage() to clone before send - this will ensure a non referenced msg is sent (and your travelling msg should be unaffected)
Will be interesting to know if this solves your issue (and if node-red succeeds in processing your data at 2khz) - please keep us updated
Yes it's possible that it is just because of my system capabilities. I'm no expert in this field I get a "RequestTimedOutError: Request timed out" error after about 10 seconds of sending 2000 objects with 4 measurements per second to the Influx-node. In my opinion, it is a huge amount of data, which I want to reduce before sending it to a database, even if the node would be capable of processing that much data.
step by step
