Only in a single user frame of reference. And actually not even then with some more complex UI's. However, I think that your point stands.
But I don't think Nick is disagreeing with that point. I regularly use globals to support state. But not flow/message state. I would use a global for an external flag that changes independently of a flow. Or I would use it as a lookup. This is not the same as what Nick is referring to which is per-message state.
Is there, I wonder, a case to be made for some mechanism to be able to force a section of flow to work in the old way? Perhaps like being able to use an option in a sub-flow?
The more I follow this thread, the more I think there are some misconceptions regarding asynchronous code and asynchronous flow execution.
I try to explain it this way:
You should never see a flow as one atomic operation, even if it does behave this way today in the absence of branches and async operations.
A flow just defines the order in which the nodes will be executed by its wires. There is no guarantee, when a node will actually be executed. That order is defined by Nick's current implementation of how he traverses the graph in the flow engine.
The second thing is the NodeJS event loop:
It processes all user code. Currently, in the case of NO branches and NO async ops, one flow gets processed as one task in the event loop. (depth-first traversal)
Now in v1.0.0 this operation will always be split up in many separate tasks, that get executed by the event loop eventually. That gives NodeJS the chance to process other tasks in the meanwhile as well.
The only guarantee is that your nodes will execute in the correct order (defined by its wiring), but other operations (nodes of other flows, branches, callbacks, etc) can run in between, too. So the chance to block the event loop by one large flow that would be executed as one chunk in <1.0.0 is greatly reduced.
Ouch. That's good to know, even if it may soon be academic.
I'm sure I can live with the new regime, even though it strikes me (comment, not criticism) that the change sort of exposes the underlying node.js infrastructure in a way that makes NR itself seem less like (my intuitive idea of) a data-flow language and more like JavaScript. (Not very clear, but I hope you get my meaning.)
That is correct, or at least becoming more so, but I feel forced to ask why? or is it necessary? Reducing blocking or enabling distributed execution are possible benefits, but being able to do one thing at a time can also have advantages.
Because a flow has never been advertised as a single atomic operation.
The fact that some nodes are synchronous and some are asynchronous in how they handle messages means users are already exposed to those inconsistencies and have to deal with them.
Making the message passing asynchronous means every node is the same regardless of its internal implementation. It means a user doesn't need to learn the difference between sync and async like they do today. It means there is one behaviour to understand.
I feel like I'm missing something here... are you saying that node-red is meant to ONLY provide inter-flow consistency by message passing? What about global and flow variables in multi-node processes? It would seem like if I ever had to grab, modify and store back for any global or flow variable, I have an INHERENT race condition.
I would think this basically makes global and flow variables unusable, at least for what I've normally seen them used for, and you'd have to move ALL state handling completely out of node-red and implement and entire message passing rectification flow for every state modification (i.e., lock, run through transform, store, release lock, or a grab latest, transform, lock and check if still latest, update, release lock type flow).
Essentially, unless Im completely misunderstanding... this kills reliable state management in node-red?
Understood. It was just convenient to be able to think of them this way, at the risk of getting burned once in while. My experience with tools like LabView and ExtendSim pushes me in the direction of thinking of synchronous execution as the default and async as the exception. Adjustments will be made.
I myself think that asyncrone flow processing brings more benefits.
It is right that be per message state should be held as part of the message and it is right that global variables should be avoided.
Global data is needed whenever information needs to be stored across messages. Example count of certain messages. In my case it is statistical data about certain arriving messages (count by type, name, calculations with data of different messages, etc ...).
The only reason why I have this done with different nodes was a better overview of what is going on versus having all in one complex function node.
I have communicated my example here to draw attention, because I think that others will also fall into this trap. My hope is more in the direction that the documentation may possibly go into, so that others may become aware of it.
So could an indication of where the usage of context is explained.
If you grab the value in the first node, and modify it in the next, that could happen. But that can happen in the current version as well if you branch a flow.
As I said, when using a global state (tied to NodeJS in general), you have to be careful. Atomic operation is only guaranteed, if that code runs as one task in the event loop.
I think the coming change is the right choice, because it uses the event loop more efficiently, embracing the nature of NodeJS and using it correctly.
I understand the benefits, I think node-red just lost a valuable feature though because global and flow variables seem to have just taken a big hit. Im not an FP advocate so I won't argue the finer points of global state management, but if the goal here is to move node-red to a "pure functional" approach, then I don't get what the global and flow contexts are even doing here anymore. If the intent is to handle only pure message passing approaches, then you HAVE to move state management out of node-red completely for reliability I think? Im not even sure that's going to be reliable actually without transactions / locks in place to make that state management atomic in some other way.
Yes, before you could only expect consistency in a fetch, branch, mutate, store flow in ONE branch... but you COULD expect consistency IN ONE BRANCH. Now you can't do that at all, right? There is an INHERENT race condition here with this approach.
Could you have a "lock" node set that you could put at the beginning and end of a state management flow to ensure single entrance or something maybe? Then you could keep your async by default approach, but allow for per flow enforcement of single entry, which might keep global and flow variable multi-node mutations in the game?
You could only expect consistency if you only used purely synchronous set of nodes, something you have no way of knowing about without testing an observing behaviour.
That is an idea I'm kicking around in my head at the moment. Need to think a bit more about how that would work.
Hi, while I have an understanding of how to deal with async code as a seasoned computer programmer, let me give an opinion as a controls engineer / industrial automation person / PLC programmer:
As time goes on more and more people from my world will be using Node-RED, and they will have the expectation of the synchronous execution because that is how IEC 61131-3 function block diagram (FBD) works -- if at all possible, it would be nice to always keep the option for the "old" behavior.
@netwingduck and @seth350: IIRC you both work in the industrial automation world, what do you think?
Edit: forgot to say "keep the option for the old behavior"
Thats my gut reaction too. I know I'd hate trying to explain to non programmer people what an async flow is / what a race condition is. I imagine you'd just get blank stares. I think the default mentality is synchronous. There's a reason every async implementation the last 5 years has moved from callback hell, to Promises, to full async/await kind of approaches: its WAY easier to reason about (even you do still have to kind of understand whats happening underneath). That said I have no idea how to do that with node-red.
As a developer, I GET why this is where its trying to go. As a user, it feels like leaking the underlying tech stack to users that aren't going to understand why they just can't use it the intuitive way.
I don't understand the issue here. Provided the complete read/modify/write sequence is performed within a node (and there are no asynchronous actions in that code) and one always bases the action to take on the current value of the context then there should be no problem. Can you provide an example of the sort of issues you envisage?
But I was thinking of it in terms of existing Node-RED capabilities - e.g. an option for sub-flows. Where you turn on the flag to get a more synchronous approach (though maybe no more so than the current approach <v1).
That would perhaps give an "out" for flows that break under the new approach?
I agree, by the way and for what its worth - that this is the right move and I suspect that the actual impact will be less than people think except in certain limited circumstances.
Sure. I have two flows that need to communicate state between each other.
Easy example, if you wanted to take all MQTT state topics and store their last known state in a global dictionary so that you can grab "last known state" at any time anywhere in your other automation flows to make decisions about current system state. If that MQTT IN node gets a bunch of publishes at the same time, and each one goes to a SINGLE function node that gets global var, adds / changes state, and stores back, you're kosher.
But as soon as you want to do something that spreads that mutation operation across multiple nodes in that flow, you are going to have race conditions modifying that state as different MQTT messages come in, right? Which makes that global state unreliable to the point that... I just don't think most non-programmers are going to understand at all. Even just sitting here thinking about it, Im not sure I can come up with a good way to do that without explicitly having some way to ensure single entrance into the flow (i.e., locks).
I.e., "Provided the complete read/modify/write sequence is performed within a node" is probably the most common way you'd see the state changes happen, but we'd be limited to ONLY doing that for reliability. That's going to cause a lot of surprised and frustrated people I think trying to build more complicated flows. If you have to explain the highlights of the functional programming paradigm to Joe Shmo before they can start using node-red... I just am hesitant to see that as a good approach...
Again, I think I can live with this, and I get the benefits.Ill be rewriting a few things to make them less modular because now things HAVE to be done in a single node context, but I can think of a few places its gonna get tricky.
Edit: To be clear the reason I think it would be MOST surprising is that node-red kinda encourages the idea of chaining nodes together to accomplish processing something... but now with the BIG CAVEAT that if you are pulling state, and handing through several processing nodes before storing back, thats basically NOT a good idea. Seems to go against the grain.
Am I right in thinking that you agree that provided one does not read the context var in one node,then modify and write it in a different node then there is no problem? If so then consider that even with the current method one cannot rely on any node one has not examined the code for being synchronous, including core nodes, and cannot guarantee that a new version of a node will not introduce an asynchronous component, then already one could have a problem with read modify write across a sequence of nodes.
Also even with a sequence of synchronous nodes if one moved one of the joining wires to a debug node for testing, then added a new wire to replace the old one, but leaving the debug node wired in, then I believe that will introduce an asynchronous break in the flow, so that would break an existing working flow.
Yup I agree. And apparently there's more issues with the current model than I'm aware of... and I have some debug nodes I need to go delete. At least we're all reliably broken the SAME way with the new approach
I just hope we move from that to a reliable WORK AROUND as well, cause being bound to a single node context really takes a bite out of the state options we have. It almost sounds like it puts you in a world where its only really good for stateless pipeline processing... If a whole new approach to state management is needed, that's a difficult problem to solve for...
I know in particular this is gonna have some interesting caveats now for home automation / industrial automation, etc.
Actually this would effect ANYTHING that read and persists across multiple nodes wouldn't it? Like if you're persisting things to disk, DBs, really anything, don't you now have race conditions through ALL of those types of flows, not just global and flow vars?
