Soliciting some architecture insight so I start off the best track before I start coding...
My Cerbo GX device is running Node Red. I also want to store and chart data, but don't want Node Red to do all that heavy lifting in addition to running the flows...
So I'm going to use an RPi5 that is on the same network as the Cerbo to store the data and produce the nice logs and graphs and such.
What is the best way, to not make performance suffer, to get the data from the Node Red server onto the RPi? Should I just have a Node Red flow that sends JSON data to a location on the Pi? Or have a Node Red flow store it into a database on the Pi? Or have Node Red broadcast MQTT data that the Pi subscribes to?
Once I get the data on the Pi, I'll look into a way to graph it. Grafana perhaps.
MQTT is a very lightweight mechanism for transferring data between devices.
It may very well be that transferring data over MQTT has less performance impact on the Cerbo GX than installing eg an Influxdb node to write to a database on the Pi.
As a bonus, you can have guaranteed delivery of MQTT messages in the event that the Pi or network is down (there is an excellent example flow somewhere on this forum)
The node @colinl/node-red-guaranteed-delivery includes an example flow for that. It guarantees that all data will be published, queuing messages up if the broker is not accessible.
Right on. I have very limited experience with MQTT. I have programmed some linux to grab some from the Cerbo GX which also broadcasts MQTT. There was some kind of keep alive that was required to make it work if I remember. Not sure if it makes more sense for me use the Cerbo GX to broadcast MQTT or Node Red. Since I've seen Node Red become very sluggish (I have no idea if that only affects the browser access to NR or if it also affects the flows themselves), I may just leave NR out of it.
I have Node Red installed to run flows so it will run regardless. The question is whether I should also have Node Red handle the MQTT or just let the Venus OS system on the Cerbo do that.
Assuming that you are not sending thousands of records per second then the overheads of MQTT in node-red will be trivial. If you are having other problems with node-red then I suggest getting those sorted first
I was having an issue getting the UI to load up in Node Red but I did get it sorted (at least for now).
I had only one chart, but I have a dozen or so flows and a bunch of Nodes and I'm sure whatever I made is far from best practices. I set a bunch of Nodes to only trigger on change and that has so far resolved the issue after I rebooted the Cerbo GX a couple of times.
What I continue to wonder is that if NR is sluggish when I try to load the UI (and it wouldn't load, it was so slow it would have taken an hour or more to load up) -- are the flows affected similarly? Or is it just the UI? I don't know and I don't see any easy way to monitor resource usage of Node Red to see trends.
Here is what I have running, probably lots of bad work on my part, but at this point everything seems to be running smooth and it has greatly reduced my workflow off grid with my generator. flows.json (133.6 KB)
Does the device have a command window where you can run commands such as top to monitor the CPU and memory usage? If so then if it is going slow run top and see if anything is clogging up the system.
Were you running the browser on the device or on a PC?
I'm running the browser on a PC that is on the same LAN as the Cerbo GX.
I haven't seen a command window with the Cerbo but it is Linux based and I use SSH to grab some files from it so I'm sure there is a way for me to monitor the CPU. I'll do some research on it.
I typically try to avoid messing with it because I'm not smart on linux and this thing is the brains of our power production off grid in Alaska in the winter far from people. What I'm trying to say is, I'm terrified of messing it up...
If it is that important a spare in the office running the flows with a simulation of the rest of the system sounds as if it would be a good investment.
I have a spare Cerbo that isn't running Node Red that I could plug in if need be. I think I'll have my Pi alert me if it can't get ahold of Node Red after X minutes so that I can investigate and swap the Cerbo out if need be. But if I can learn best practices and tighten up my flows I'm sure that won't be required. I just need to do more research.
Some questions I have:
a) instead of having a dozen flows, if I put all the logic onto one flow, would that save performance? I'm guessing not.
b) related, if I have one node firing off on one flow every 2 seconds which triggers logic, and another flow with the same node firing every 2 seconds triggering other logic, would it be better for performance if I moved the logic all onto one flow and had it all fall under one single instance of that node firing every 2 seconds? I'm guessing this would be better.
I have no doubt Node Red will continue to do great so long as I don't mess it up with my sloppy designs.
I don't want to discourage you from considering efficiency in your design, but my feeling is that having your code in well defined and discrete chunks - ie a logic flow, business process, whatever you call it, should ideally have it's own editor tab, not be mingled together with other flows.
If you do end up with one flow which looks sloppy or underperforms, keeping them apart makes it much easier to rebuild just that flow.
Bear in mind that a message will pass along the wires from node to node in milliseconds.
Something which happens every 2 seconds is probably not an important candidate for optimisation at this stage.
I know nothing about the Cerbo, but you have masses of computer power in a Pi 5.
No, it won't make any noticable difference. It is what your flows are actually doing that will be consuming resources, not the node-red overheads of passing messages about.
If you are only doing anything every two seconds then, unless whatever you are doing uses a lot of processor resources, you should not be loading the processor, unless it is a very low power device. Do you know how much RAM it has and how the cpu performance compares, for example, to a Pi Zero?
Experiment with that to find how to run top or similar command to monitor performance. Once you know how to do that then running it on the working system will tell you a lot about what is going on.
The Cerbo GX has a dual core CPU and 1GB of RAM. Not sure how that compares to the Pi Zero though.
Ever since getting rid of my charts I haven't had any issue with Node Red bogging down. I'll experiment with top on the Cerbo like you suggest and see what I can learn.
