Handling variables through the flow

General
1

Hi,

Am new here, have had a search but forgive me if this has been asked before but could not find the answer.

Have installed NodeRed on a Pi that will be receiving data from a GPIO connected RF hat device, the data comes in on the serial port is processed and then makes updates to a MySQL database installed locally. The data from the serial port is broken down and stored in flow variables and process through the flow. All is working well,

Have now implemented an MQTT branch that will send the same serial data to a cloud server that has the same NodeRed flow and MySQL database installed on it. All working well.

Am having a debate with a colleague, I appreciate you can make nodes synchronous or asynchronous but don't know how the whole flow is managed/controlled.

The query is with the cloud based flow, for example if we installed hundreds/thousands of Pi's that were sending data to NodeRed via the MQTT, does the flow complete before the next one starts, or if the data is coming in quicker than the flow completes does that mean that multiple flows are active at the same time? If the latter, are the flow variables exclusive to the flow instance, or could the possibility exist where the flow variables are shared over multiple instances meaning you could get unexpected results as one flow is changing the variable values in the other active flows?

I hope that makes sense, and apologies if this 'simple' question turned into a bit of an essay, but could think of a more concise way of describing the question!

Thanks in advance for any help or input :slight_smile:

WDL

2

If you mean by flow variable, message properties, then they exist in the nodes they pass through and new calls to that flow have separate message variables. But are you meaning flow context variable, which exist for any flow in the tab and can cause concurrency issues.

3

Please don't do it that way. Keep the data in the messages as it flows through your nodes, then when you get to writing it to the database you know that it is self consistent.

No, when a node processes a message and passes it on down the wire, then that message will be queued at the input to the next node, and another node, which could be anywhere in the system, will get given the processor to service a message that is waiting to be processed on its input. A round robin approach is used so that all nodes with waiting messages get the processor in turn. The result is that there can be many messages passing down a flow one behind the other. That is why flow and global context should be avoided wherever practical, as it is very easy to get race conditions. Keep the data in the messages. A common approach, when one has something that needs to be retained to be used further down the flow, is to move it into a property in the message. Generally nodes will pass through messages, leaving properties that they do not know about unaffected, so that they can be picked up again later down the flow.

4

Hi E1cid,

Thanks for the reply.

I receive a CSV string on the MQTT, the message is broken down and assigned to flow.set variables so that they are accessible throughout the flow and various processing performed on them. IF two flows can exist at the same time, are these variable values in scope across the flows, so if they are changed in one, are they changed across all live instances.

Just occurred to me that I can test this, can inject messages into a flow that is deliberately slowed down, change the variable value down the flow and see if new messages are 'held' or processed asynchronously and if changing the flow.set variables change the values in other instances.

WDL

5

Node-red is single threaded, so only one node is being processed at any instant.

However if a node is waiting for some external process eg updating a database, Node-red is likely to use the time to process another node.

As @Colin says, it is a mistake to copy message properties into context variables. They are already in the message properties which get passed from node to node and are unique to that message. Write your code to use these message properties.

That's really bad Node-red form. You are setting yourself up for data integrity problems

6

Thanks Colin, just picking up a NodeRed project from someone else, so a new concept and also not that familiar to js, but have found the transition on the language front easy enough.

Am impressed in how it works on both Pi and web sever as been easy to implement, just trying to understand and mitigate any data issues down the line if the project scales.

WDL

7

Thanks jbudd,
Will go through all the replies in a little more detail tomorrow, many thanks for your reply.

8

Hi,

I should have done this originally, this is what I have been doing with the data packet when it first arrives;

var oData = msg.payload.split(",");
flow.set("Header", oData[0]);
flow.set("Type", oData[1]);
flow.set("Instruction", oData[2]);
flow.set("Index", oData[3]);
flow.set("Parameter", oData[4]);

and then referencing these further down the flow in various nodes;

if (flow.get("Type") == 1 || flow.get("Type") == 2) {
    msg.topic = "Update Name";
    return msg;
}

From reading the replies, am looking at replacing this with;

    msg.oPacket = {
            header : oData[0]),
            type : Data[1]),
            instruction : oData[2]),
            index : oData[3]),
            parameter : oData[4]),
    };

I assume this is better as the variables are in the msg and will be static for the duration of the flow instance?

I would then use these further down the flow;

if (msg.oPacket.Type == 1 || msg.oPacket.Type == 2) {
    msg.topic = "Update Name";
    return msg;
}

Hope I have interpreted this correctly.

WDL

3 Likes
9

Yes, much better (and safer). Much more akin to a PURE approach.

Context (flow/global) can be useful but they should most definitely not be the goto/first option. When and where context is right/sensible will come with time.

10

Just to add:

There are some nodes that do not respect this approach (i.e. they return a new/blank msg). I always recommend you notify the node author of this and/or move to an alternative node package.

As always, the very helpful folk on this forum can advise you when you hit one of these issues.

11

Just to clarify, if you are talking about Flows and not processes in a Flow then no, flow context is only 'in scope' to one Flow so other Flows, even if identical, will not be affected.

12

Where, in this context, a flow is a tab in the node red editor.

1 Like
13

I see what you did there :laughing:

1 Like
14

Getting used to the terminology of NR, I saw a flow as a chain, which there could be multiples of within a tab, but if the contents of a tab is describes as a flow, have learnt something :slightly_smiling_face:

Contextually yours,

WDL

1 Like