Hi all,
Hoping for some guidance with a issue I'm having with the RevPi running Node-Red.
Currently I have the RevPi communicating to a Allen Bradley Compact Logix PLC via Ethernet connection, and have setup a timestamp in milliseconds to constantly update to the PLC.
The problem is that at regular time intervals, the PLC tag stops updating for about 400ms. While in top or htop, i can see that the instance for Node-Red spikes the CPU quite high, and these spike correlate with the comms 'stutter' I'm experiencing at the other end.
Has anyone experienced something like this before? I'm not sure what the root cause of the issue is, testing so far hasn't yielded any obvious results.
What else are you running?
I often get significant spikes on my Pi 2 that come mainly from InfluxDB (I really need to trim the db a little, it is getting too large).
You might also get problems like this if you are hitting some swap usage. If you are using SD-cards for main storage, they are very slow and so if something gets swapped out/in, you may see a noticeable pause.
Also remember that Linux is not a real-time OS so spikes and pauses are not uncommon.
Thanks for the speedy reply.
It is a fresh install so shouldn't be running anything else. Also seems strange that the Node-Red instance is the one that spikes the CPU, not any other process.
The storage is on-board as its a RevPi, I'll look into that but i think it should be decently fast.
The problem is always at the exact same time interval apart, there are smaller stutters inbetween and then bigger stutters about 12 seconds apart. This time interval also shortens if i clog the CPU up with more logic to run.
Hmm i was worried it may be an inherent Pi/Linux thing. D=
Are you running quite a high message rate ? or large objects ? Node.js (on which Node-RED runs) does do garbage collection as and when it thinks it is running out of memory - On a low powered device the garbage collection can be significant. You can set the memory limit at which GC kicks in so you may be able to tune it somewhat depending on your device. But it will mainly depend on the profile of your application.
I was hoping for a polling rate of 10ms, but even changing that to 250ms doesn't seem to make any difference (apart from less average CPU usage). The message is a single Int, the project needs 6 of these but right now i'm only testing with one.
I also tried changing the memory allocation with Modify Environment="NODE_OPTIONS=--max_old_space_size=256" to Environment="NODE_OPTIONS=--max_old_space_size=512"
but didn't make any noticeable change.
Also side note i tried the exact same build with a spare regular rPi from home, and get the same reults/issues
Alright great i hadn't heard of garbage collection before, i'll have a look into that now.
Thanks for the insight!
6 readings every 10mS does sound a bit fast fro a Pi level device - but certainly 100mS (or 250) should be fine. What processing is occurring per msg after that ? If it's "just" sending out over ethernet that should be fine (vs database or web access for example) - though again the ethernet on a normal Pi is shared with the USB controller and so there are interactions there also.
Ideally it would be 10ms, but was going to tune and set it to just below whatever i start getting issues.
For the actual project/code i'm receiving a buffer via a udp connection, then running that through a function node that isolates the two bytes that i want, and packages them into an Int for sending to the PLC.
Unfortunately even with all that code disabled, and only a simple timestamp into a function (to get the milliseconds) and out to an EthIP node i'm getting the same lockups.
Nothing else running on the USB ports on this unit, thought i haven't disabled anything to do with the USB ports yet. Might be next on my list.
For reference, the trend from the PLC capturing the comms lockups:
You could try turning on trace level logging in Node-RED to see if that gives you any clues. Also try removing the debug node to see if that helps.
As a worse case, you might try temporarily removing Node-RED and using a dedicated Node.js app to see if that removes the issue. If it does, you might be better off using the app for the comms and Node-RED for other processing.
Have you fully disabled (or not installed) the GUI on the pi? That will release resources.
@TotallyInformation
Alright awesome, sounds like i have a few fault finding methods to try, thanks again for the help.
@Colin Yeah disabled the GUI already hoping that might have been the cause.
I'll be looking into this sporadically as i have another project to complete, but when i get aorund to more testing i'll post the results up (or hopefully the solution!).