Factors affecting speed of Node Red. Close to real-time capabilities?

[lizhong]

HI all, I'm currently working on a project with a few sensors that feeds me real time data at an interval of 1ms. By using node-red on my Pi 2 Model B (900MHz chip) as a way of control, I obtain my sensor data together with the timestamp via the inject node & write it into my influxDB server. The inject node is being run on an interval of 1ms as I require as much data as possible for visualization and analysis. From what I noticed, the timestamp when I query my data from influxDB, it has sometimes even a margin of error of 10ms & rarely the 1ms interval that I wanted. Wondering if there are any possibilities to increase the speed of Node Red itself. Should I maybe update my hardware to a Pi4 with a great SD Card? what are the other factors that affect node red?

Am aware of similar topics found on this forum.

How fast is Node-Red?

Hi Folks, I am learning Node-Red and already have implemented together with influxDB and Grafana for basic data acquisition purposes. I would like to also make a vehicle data logger, which require high resolution data and faster acquisition speeds (~300 sps) from CAN BUS and different acceleration sensors etc. Now the question is, how fast Node-Red can be? I am interested in application particularly on Raspberry Pi. Is it also suitable for bandwidth heavy applications, or only for "low sample …

Floating point calculation speed

You probably should also avoid repeated calls to expensive math functions. For example, in the X-axis rotation matrix, var cosx = Math.cos(Xrot); var sinx = Math.sin(Xrot); var Y1 = l1y*cosx-l1z*sinx; var Z1 = l1z*cosx+l1y*sinx; could be up to twice as fast as what you are doing now. It can be tempting (and more readable) to just code up the textbook equations, but if performance is an issue, think like a computer not a human.

Just wondering if there are many more ppl that tried using Node red for such Real time applications and what were their experiences.

Thank you so much for your help in advance

[TotallyInformation]

Firstly, Linux is not a real-time OS, it is a general purpose compute platform.

Secondly, Node.js (that underpins Node-RED) is largely single threaded. And it emulates multi-threading by using async processing (which can/will take many "ticks" to complete).

Thirdly, Node-RED is a general purpose, low-code, development platform (e.g. not specifically designed for real-time).

Fourth, Node.js is a JIT compiled language and FAR from being "fast" in the sense of C-based, Rust or assembler languages.

So all 4 of these things conspire against you when you try to push the limits down to single ms responses. In truth, I very much doubt any hardware platform is going to be able to give single ms reliable responses with this stack let alone a cheap education focused platform like the Pi, even more especially one as old as a Pi2.

InfluxDB will add its own issues. By default, it can record down to nanosecond resolution but that doesn't mean that it can cope with incoming data reliably at even ms precision because to do so, it has to have reliable uninterrupted access to RAM. As soon as your data exceeds RAM size, you are forced to rely on disk I/O performance. InfluxDB does its best to minimise any such issues but physics will always win out eventually.

If you really need a real-time system, you need to be using a real-time OS and suitably tuned hardware.

At least, you would likely be better off using a dedicate C++/Rust program to pass data direct from sensors to InfluxDB and to use Node-RED to handle other aspects of the processing that are less time sensitive.

It would also be interesting to understand what process you are trying to monitor that you think needs ms precision. Not denying that there are plenty of possible examples but they are probably fewer than some might imagine.

[lizhong]

Thank you very much for your time for this reply. I need such precision due to interest in observing the raw AC current coming from my transformer sensor. Guess I might have to send my sensor data in packets instead or maybe use a buffer.

[Steve-Mcl]

If you have the ability to buffer 1000 samples (for example) then inserting to your database every 1s should be a breeze.

[TotallyInformation]

Just remember to capture the timestamp in this case as you will need that to feed into your db.

[lizhong]

Thanks for the advice & reminder. I do get my timestamps together with each individual sensor reading. Will just have to make sure how to increment them based on the time I write into my db server

[lizhong]

I do plan to insert into my database every 20ms. I am receiving almost 50k samples per second, hence probably best to write into the database at a shorter time interval.

[Colin]

@lizhong

I do get my timestamps together with each individual sensor reading. Will just have to make sure how to increment them based on the time I write into my db server

Why would you increment them? If each sample has a timestamp then that timestamp should be written with that sample. Buffering samples does not mean that each one does not have its own timestamp, just that you write all the records with one call to the database.