I have several flows running on and RPI that are keeping counts for various systems. One group of these systems is updated every hour.
For the sake of compartmentalization each of these machines has its own flow, but all these machines flows follow the same structure. Basically synchronize on start, increment count, and at hour interval transmit the api request for each individually.
Each of the flows is using context flow variables, but I’m wondering if I would be better off using global variables? Then I could transmit all the post objects at one time, instead of doing 10 individual ones consecutively.
If you get rid of the context variables and pass all the data around in the messages (which is the node-red way) then possibly you can use just one flow and send it messages to handle the individual machines.
Please don't do that!
Global context variables are An Invention Of The Devil.
One problem is that you can break a completely unrelated flow if in a new flow you happen to pick the same variable name and change it's value.
Context variables are useful when:
A node needs to compare a property the current message with a previous message (node context)
Two bits of the same editor tab which are not connected by wires need to reference the same data (flow context)
A constant needs to be stored globally and may be referenced in many places - eg an API key (global context, though this is arguably better done with environment variables)
As @Colin says, Node-red provides the message object to pass data from one node to another. Take advantage of this and create a message property instead of a context variable.
Thanks for the feedback, will have to think about how I would accomplish this.
I am incrementing my counter values for all 10 about every second or so, and that’s why I am storing in the flow variables. Every hour I execute the API request for each, but the values are changing basically all the time it the equipment is in operation.
Is the start operation under node-red control or are you reading this from the machine?
If node-red is keeping track of the cycles then you certainly need somewhere to store that. Are you using persistent context for that? Probably a database would be more conventional.
The machines are standalone devices, and I just read their counter values or in some cases just look for a signal transition to determine if a cycle has occurred.
I would like to set this up as persistent, but haven’t had the time to figure that out yet.
I also keep track of hour meters for a different category of machines, but it’s done a little differently. Hour meters are stored in a central plc, and I just update them once daily. The value is then posted through api. This group is on a different schedule.
Well, some people have tried to put you off using globals. However, these may in fact be ideal. Since you can easily persist context variables.
As long as you bear these things in mind:
The sum of all of the variables need to fit comfortably into memory along side everything else.
As others have said, you must be careful what updates the variables. This should be pretty simple in your use-case though.
Be careful if persisting to file as the values are only written every 30sec unless you adjust the default write-through time.
Context variables are simple to use, easily persisted and are a very viable alternative to using a database in these limited cases.
To help with flow complexity, I would recommend sorting through your flow sections, wrap them in a group and top-and-tail them with a link-in/link-out where the out uses the "Return to calling link node" setting. With that, you can now have common compute flows that you pass data into from anywhere you need to and that returns the output. Like a sub-flow but without the overheads. Though subflows might also work for you. (you may want to add a msg._from to messages if you start to lose track of things, a msg._updated timestamp can also sometimes be helpful.)
In my view, for any reasonably simple data, using context variables is a perfectly good alternative to using a database if you don't need the capabilities that a database engine brings with it.
I have some very large JSON datasets in my live home automation server such as around 10 years worth of daily npm download stats for my main repositories. I really don't need to do much with them so no need for the complexity of a database just for those. Performance is great, simple to use should I need to, data is already at hand.
Julian makes a valid point that it is easy to persist context variables.
However, if it is important that your data persists over restarts, I find the context store method involving a scattering of json files and 30 second update interval very ragtailed.
Undoubtedly it does work but I would not call it robust.
Persistence is somewhat important as these counts is how we are generating preventative maintenance work orders. With that said, because of how frequently I am posting to the API if I lose power (which it’s connected to a ups also) if I lost 3600 counts for an interval that happens every 500k, it’s not the end of the world.
The most I would have is an hour of lost counts. When I start it up, I synchronize my flow variable with the value from the last api post before I continue counting.
Thanks for all the feedback and information, really appreciate it.
We ended up making a backup system entirely using context Basically we send a lot of telemetry to cloud db, but as soon as a request fails, it is added to a long queue in context. Every 5 min there is a flow to read and try to resend failed requests. If successful, it is deleted from context. If not, it remains, until eventually old junk is deleted after a set limit (24 hours).
I would suggest to only optimize anything when it is needed (either because of performance issues, or because the current system is hard to work with).
If your server can deal with the current architecture and there is enough network bandwidth, I would not complicate my life with manual data synchronization, global variables, single network call, etc., because the end results seems much harder to work with and easier to mess up / introduce bugs.
But since you mentioned each machine has a dedicated flow, I would rather try to simplify that (just like in programming: DRY principle) and implement the generic flow only once and call it with different parameters via link in/out+call nodes.
Basically we send a lot of telemetry to cloud db, but as soon as a request fails, it is added to a long queue in context. Every 5 min there is a flow to read and try to resend failed requests. If successful, it is deleted from context. If not, it remains, until eventually old junk is deleted after a set limit (24 hours).