My (new) approach to managing state (encapsulation using groups)

[prof7bit]

I am using Node-RED exclusively for home automation, specifically as an add-on to Home Assistant.

In this use case I am not so much dealing with complex contents of messages and their transformation, but rather with the routing of simple messages where each message signifies the triggering of a sensor, a switch, etc., often the routing depends on states of other switches, certain time windows or also the order in which certain messages arrived in a certain time window. For example If first the motion sensor in the driveway triggers, then soon after the sensor in the carport, I want different lights go on than when this happens in the opposite order.

For this I find myself making use of flow variables on almost every flow I create to manage that state.

And while working with flow variables I soon identified these sources of error:

• documentation (naming) of nodes not reflecting or (even worse) out of sync with what side effects it actually has
• hard to understand (or even see) the side effects when looking at the flow

Consider this:

A function node named "set time_a" and containing the code "flow.set('time_entrance', ...);". Obviously at one time in the past, in a hurry, I renamed the flow variable and forgot to update the node name. Or I update it in two places but it is also used in one more place I have totally forgotten.

---

So I thought by myself there must be some way to make this more transparent and less error prone. I came up with following solutions:

• dynamically parse the name of the flow variable (and other suff I normally would have wanted to hardcode) out of 'node.name', NEVER hardcode any names of flow variables in the code: Consider a function node is called "foo_state = maybe_arriving", it might contain the following code:

const name = node.name.split(' ')[0];
const value = node.name.split(' ')[2];
flow.set(name, value);
return msg;

• confine any usage state to very few and easily identifiable nodes. Often one node is setting the state and then some time later a different message coming along through a different node is reading it, put both nodes next to each other and draw a group rectangle around it. Never access the state outside the group, instead after reading it attach it to the message, so it can be used in later nodes and switches, thereby keeping all of the flow outside the group rectangles pure.

• do not manually assign flow variable names at all! Use the NR_GROUP_ID to generate a name for the variable inside the group, so you cannot accidentally interfere with it from outside the group, even if you wanted to. Encapsulate all flow state in groups. This has the added benefit that you can just copy paste the stateful group without needing to rename variables.

The store/recall group implements both of the above suggestions: group ID and parsing of node name:

[{"id":"5da5765352833db3","type":"group","z":"27ec49e7bab0eeaa","name":"Store / Recall","style":{"label":true},"nodes":["e164128c86ef4ffd","09c23ec1fd7d8ea6"],"x":294,"y":359,"w":172,"h":122},{"id":"e164128c86ef4ffd","type":"function","z":"27ec49e7bab0eeaa","g":"5da5765352833db3","name":"store","func":"const name = env.get('NR_GROUP_ID');\nflow.set(name, msg);\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":370,"y":400,"wires":[["1bae87243139671c"]]},{"id":"09c23ec1fd7d8ea6","type":"function","z":"27ec49e7bab0eeaa","g":"5da5765352833db3","name":"recall a","func":"const name = env.get('NR_GROUP_ID');\nconst recall_name = node.name.split(' ')[1];\nmsg[recall_name] = flow.get(name);\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":380,"y":440,"wires":[["12ca9370b832f4aa"]]}]

The "rate" node below then just uses msg.a and msg.b to do its calculations.

Another example for a reusable stateful group: the MonoFlop Switch that is going to be used in my new implementation of my outside motion sensor light control:

[{"id":"099d2de186307592","type":"inject","z":"3ec3298160bac448","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":200,"y":140,"wires":[["f94abb65b10847fd"]]},{"id":"c62bc0d2c0d7c367","type":"debug","z":"3ec3298160bac448","name":"debug 13","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":760,"y":200,"wires":[]},{"id":"3e1a547c0878192d","type":"inject","z":"3ec3298160bac448","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":200,"y":220,"wires":[["767058d4b6a1b3ac"]]},{"id":"631dee1cc49cc6cd","type":"group","z":"3ec3298160bac448","name":"MonoFlop Switch","style":{"label":true},"nodes":["f94abb65b10847fd","767058d4b6a1b3ac","6afe9bdcca906fe2","267e1eba2bb3419d"],"x":334,"y":79,"w":312,"h":182},{"id":"f94abb65b10847fd","type":"trigger","z":"3ec3298160bac448","g":"631dee1cc49cc6cd","name":"","op1":"","op2":"","op1type":"pay","op2type":"payl","duration":"6","extend":true,"overrideDelay":false,"units":"s","reset":"","bytopic":"all","topic":"topic","outputs":2,"x":420,"y":140,"wires":[["6afe9bdcca906fe2"],["267e1eba2bb3419d"]]},{"id":"767058d4b6a1b3ac","type":"switch","z":"3ec3298160bac448","g":"631dee1cc49cc6cd","name":"","property":"${NR_GROUP_ID}","propertyType":"flow","rules":[{"t":"true"},{"t":"else"}],"checkall":"true","repair":false,"outputs":2,"x":410,"y":220,"wires":[["c62bc0d2c0d7c367"],[]]},{"id":"6afe9bdcca906fe2","type":"change","z":"3ec3298160bac448","g":"631dee1cc49cc6cd","name":"true","rules":[{"t":"set","p":"${NR_GROUP_ID}","pt":"flow","to":"true","tot":"bool"}],"action":"","property":"","from":"","to":"","reg":false,"x":570,"y":120,"wires":[[]]},{"id":"267e1eba2bb3419d","type":"change","z":"3ec3298160bac448","g":"631dee1cc49cc6cd","name":"false","rules":[{"t":"set","p":"${NR_GROUP_ID}","pt":"flow","to":"false","tot":"bool"}],"action":"","property":"","from":"","to":"","reg":false,"x":570,"y":160,"wires":[[]]}]

I can copy paste this group multiple times and each instance will automatically have its own state encapsulated, impossible to interfere with each other.

---

Wish list:

• sub flows with multiple inputs
• group.get() and group.set()

[bakman2]

Correct me if I am wrong, but isn't the whole point of Home Assistant that it is a state-machine ?
ie. If you want to know the state, you ask the state.

[prof7bit]

Yes, but I often have a lot of intermediate state within my flows. For example in my garage door automation I get from Home assistant the state of the door sensor and my gps distance from home. But within my automation I have a bunch of internal states (how often has the distance been greater than x, can I trigger door opening already or was this just a gps glitch, etc).

I could of course create home assistant entities for all these internal (to my flow) states, but this would be even more complicated and instead of confining the scope of any state to be not larger than absolutely needed (best practice) I would even widen it across the entire home assistant.

For example look at the example flow where I calculate my gas flow rate, I need the state of the counter from one minute ago to calculate the volume per time. In my example I just dropped the "store/recall" group onto my flow and can instantly use it, with your suggestion I would have to come up with a name for a new virtual sensor in home assistant that has the gas counter from one minute ago, update it and read it, thereby maybe even distorting the time stamps that I need for calculating the rate.

Or I could do it like I did it in the beginning: come up with a good name for a flow variable and use flow.set()/flow.get() with that name to store that state, but that has all the problems I mentioned in post 1, I need to come up with globally unique names for things that are so trivial or so local that choosing a good name for it (and typing it out in all places you need it) becomes harder and more time consuming and annoying (and error prone) than actually using it.

My goal was to not have to think about that kind of intermediate state (and especially which name to assign it) anymore at all, I want it to be visually edited (like the rest of NR), easy to see what is connected and not needing to manually assign names for very local and transient internal states.

[ralphwetzel]

@prof7bit

For example look at the example flow where I calculate my gas flow rate, I need the state of the counter from one minute ago to calculate the volume per time.

While not arguing against the way you choose (if it works, it's fine!):
I usually ask myself one question: Is my problem so unique that I have to create my individual solution - or has someone else already solved it & contributed a node. In most scenarios, the second is the answer. Averaging over time or number of messages is one of those cases...

[prof7bit]

I usually try to avoid external dependencies for trivial stuff like this. But even without this particular example the point I'm trying to make still exists:

The actual point of this thread is that found a practicable way (or call it code pattern) to encapsulate local state for multiple nodes using a group instead of a sub flow, thereby making such a group immediately reusable without cumbersome and error prone manual renaming of variables.

The point of this thread is NOT how I differentiate my gas counter, this is just one simple example application that needs state. I choose this example because it is simple and it needs state. Like a "hello world" is supposed to be simple it still serves a purpose to show its code, nobody would suggest to just download the compiled "hello world" executable from a 3rd party contributor instead.

And unless we can have nodes with multiple inputs (which is probably not going to happen in the foreseeable future) such constructs cannot just be replaced with a contributed node or a sub flow. Groups are also much easier to debug than a sub flow.

I thought I should share it because I could not find any prior references of this technique having been used or recommended before.

[bakman2]

The actual point of this thread is that found a practicable way (or call it code pattern) to encapsulate local state for multiple nodes using a group instead of a sub flow, thereby making such a group immediately reusable without cumbersome and error prone manual renaming of variables.

To be honest I did not try the flows as I didn't understand what it was trying to solve (I am usually already confused when I see home assistant nodes from the get-go).

But trying the monoflop example I see what you are saying, it is clever and could indeed be useful at times. Not sure about the gas counter example - I would use a database instead.

I wonder, over time, how many flow variables become orphaned and the maintenance of it, how would you determine which ones are used and which aren't. There is also no initial state/error handling, ie. if flow variable does not exist, how does the flow respond ?

But I get the idea, nice thanks.

And a nice to know; variables can also contain functions, can sometimes come in handy for reusability.

[tve]

Cool ideas! I use flow variables a lot and keeping track of them is a chore and error prone, the somewhat clunky side-bar doesn't help.

I'm trying to think how your scheme could be enhanced using custom nodes. E.g., nodes can depend on config nodes and there's nothing inherent that would prevent nodes depending on other nodes, although Node-RED will likely choke on that (it might ignore it).

So what if you had a "define variable" node and a "use variable" node, and in the "use variable" you'd select the "define" node it's associated with just like you'd select a config node? That would work, but not accomplish the "no-config" goal in that if you copy-paste a pair of define&use nodes you'd have to go in and make sure you change the association. It wouldn't be automatic like when you copy-paste your groups... (And I don't think there's any way to hook into a paste operation as a whole to tweak the associations automatically.)

If I were you I would probably define some global helper functions that you can use anywhere, for example, your group.get()/.set(). Or maybe write custom nodes that do the store/recall or payload into/out of the variable.

Thanks for sharing!

[prof7bit]

@bakman2

I wonder, over time, how many flow variables become orphaned and the maintenance of it, how would you determine which ones are used and which aren't.

You identified a valid problem, I have no direct answer (yet), but I have a workaround. I currently do not use a persistent storage for the context variables, so it would somehow clean up itself automatically on the next restart, which would happen at least every time I install the next node-RED update. But this also immediately leads me to the second issue you identified...

@bakman2

There is also no initial state/error handling, ie. if flow variable does not exist, how does the flow respond ?

The MonoFlop switch (and also a similar FlipFlop switch I did not post yet) use only 2 possible states and have a condition "if true / otherwise" and this would trigger the otherwise route if it is undefined. So the switch would always start out in the default (not activated) position, which seems reasonable and expected behaviour.

A little less friendly is the Store / Recall group. the recall node would leave the target message property undefined.

I must react to this by being very defensive in my programming. The code for the rate node from the gas counter screenshot currently looks like this:

if (msg.b) {
    const time_a = new Date(msg.a.data.new_state.last_changed).getTime();
    const time_b = new Date(msg.b.data.new_state.last_changed).getTime();
    const dt = (time_a - time_b) / 1000;
    const dv = msg.a.payload - msg.b.payload;

    if (dt > 0) {
        msg.payload = 3600 * dv / dt;
    } else {
        msg.payload = 0;
    }
    return msg;
}

So if msg.b is undefined the flow stops right there until at least one counter increment has happened and one minute has passed since then.

I usually always test things like this by clearing all flow variables and then see whether the flow can handle a cold start gracefully (no lights switching on or off unexpectedly during a reboot in the middle of the night, etc). Because there will be cold starts later in real life on every reboot / restart anyway.

So once I have tested that everything behaves nicely in the absence of state variables I can then at any time remove all state variable whenever I want to clean up stale entries, and this happens only when editing the flow (removing / adding groups) anyway. So at least in my case the stale entries are not really a problem.

The only real problem I see is the impossibility to identify the variables in the sidebar, maybe one could work around this with a function node in the group that uses node.status() to show the state with status icon and/or text below the node, this could already be enough in most cases. For example the trigger node already automatically shows a blue dot when it is active, so I can quickly see whether my MonoFlop has been triggered. I could instrument all my groups in a way to show some kind of status if the existing nodes don't do it themselves.

[waldbaer]

@prof7bit Very interresting discussion!

I have in my home automation setup also a lot of states (presence, open windows, etc etc).
To overcome the problem of consistently accessing all of them (use the correct name, scope and storage type) I developed in the last months a new node bringing a user-friendly abstraction of the NodeRED context storage.

Basic idea is to have one or multiple configuration nodes where all known and used states, config options etc can be configured. In the concrete nodes to read and write to the state just a selection of the configuration and the state to be accessed needs to be configured. No more string constant etc to access the right context.
(Similar idea as described by @tve)

So in your case you could add a dedicated configuration node where all final or intermediate states of the garage are described. With concrete nodes the different states can be update, read out etc.
The node also supports an "collection of multiple state values". Just chain a set of the nodes and enable the "Collect Values" option. All collected states etc. will be added to a separate msg property as attributes. Then a custom function node could do it's control logic based on all necessary states.

I just pushed a first version of the new node-red-persistent-values node in the last days. It's not yet published to NPM but can already be installed via

npm install waldbaer/node-red-persistent-values#master

I would be happy to get some early feedback.

Best regards
Sebastian