I cant see how to node-ify my function?

General
1

I cant seem to see how to accomplish with nodes what i can do with a simple function?

Perhaps i made a bad design decision taking advantage of the context storage to maintain a list of sensors that get updated vie ble/mqtt far more frequently than required for my basic hvac and monitoring requirements.
My thinking was with a referenceable list of current sensor readings i can query for display ,testing, graphing and logging at any level of granularity required.

if triggered by incoming mqtt updates (ble advertising broadcasts) flows/tests etc may be triggered way more frequently than required, node red is hosted on a limited pi-0-w.. ?? Requiring perhaps some sort of message throttle?

However when it comes to maintaining the context list of sensor objects I cant seem to grasp how to express it with nodes.

I can supply the code if required but its basically

for ( each obj in context sensor list){
     If obj.key = msg.payload.key{
          for (each property of obj) {
               If msg.payload.property exists 
                     obj.property = msg.payload.property
           }
           update obj.lastupdated timestamp
           save object list with updated item obj
           Quit Success
     }
  }
Quit Warning-sensor not found/not an expected sensor message

er i write my own pseudo code

2

It would be handy to see an example or two of your incoming sensor readings and your [actual or proposed] context variable.

I think you can run one or two flows on a Pi 0 quite comfortably. Worry about throttling if it turns out to be a problem.

3

Hi.

I am not getting/understanding what you are wanting to do - clearly.

What I'm reading:
You have sensors that send data - via MQTT - to your flow.

But then what do you want to do?

You then go on about a context list of sensor objects.

Do you mean a way to know if the sensor is on/off line?

If that is so, you use the LWT / Birth Certificate messages.
Birth Certificate messages are sent ONCE when the device connects. Telling you the device has connected.
In a similar vein, LWT are used if the device goes off line.
(How they work is tricky and I shall explain basically here.)

The device connects.
It sends it's BC message. (On a special topic)
You can monitor this channel and if the BC message is received, you can indicate this on the dashboard (if you want)
ALSO the LWT message is sent (again, on a special topic) but nothing is done.
If the MQTT connection is broken - I won't go into that here/now - the LWT message is then broadcast telling everyone that the device has "gone away".

So, with a little bit of code you can have the device's status shown determined by the receipt of BC and LWT messages.

Screenshot from 2025-08-13 08-19-06
Screenshot from 2025-08-13 08-19-06651×675 40.9 KB

Here you can see all THREE messages.

So with that you could/would do something like:

Screenshot from 2025-08-13 08-23-42
Screenshot from 2025-08-13 08-23-421920×1080 175 KB

And the output of the function node would then indicate the device's status.

NOTE

This is a BASIC overview of what to do.
You will need to make each device's message identify itself.

But I hope that helps you with what you want to do.

4 
N: [ OMG->MQTT ] topic: home/OMG_ATOM_L/BTtoMQTT/A4C138274210 msg: {"id":"A4:C1:38:27:42:10","name":"ATC_274210","rssi":-70,"brand":"Xiaomi","model":"TH Sensor","model_id":"LYWSD03MMC/MJWSD05MMC_ATC","type":"THB","tempc":27.5,"tempf":81.5,"hum":55,"batt":93,"volt":3.045,"mac":"A4:C1:38:27:42:10"} 
NN: : BTF oDuenvdi ce20  ddeetveiccteesd,:  sA4:C1:38:46:47:C3c
a
end
W: [ WebUI ] webUIQueue full, discarding signal BTtoMQTT           46:47:C3
N: [ OMG->MQTT ] topic: home/OMG_ATOM_L/BTtoMQTT/A4C1384647C3 msg: {"id":"A4:C1:38:46:47:C3","rssi":-70,"brand":"Xiaomi","model":"TH Sensor","model_id":"LYWSD03MMC/MJWSD05MMC_ATC","type":"THB","tempc":27.4,"tempf":81.32,"hum":51,"batt":54,"volt":2.71,"mac":"A4:C1:38:46:47:C3"}N 
 E9:61:F2:12:5C:5A
N: [ OMG->MQTT ] topic: home/OMG_ATOM_L/BTtoMQTT/E961F2125C5A msg: {"id":"E9:61:F2:12:5C:5A","name":"mk.E961F2125C5A","rssi":-98} 
N: BT Device detected: A4:C1:38:48:67:AD
W: [ WebUI ] webUIQueue full, discarding signal BTtoMQTT           48:67:AD
N: [ OMG->MQTT ] Ntopic: home/OMG_ATOM_L/BTtoMQTT/A4C1384867AD msg: {"id":"A4:C1:38:48:67:AD","rssi":-83,"brand":"Xiaomi","model":"TH Sensor","model_id":"LYWSD03MMC/MJWSD05MMC_ATC","type":"THB","tempc":29.7,"tempf":85.46,"hum":45,"batt":83,"volt":2.96,"mac":"A4:C1:38:48:67:AD"} 
: 58:2D:34:38:19:97
W: [ WebUI ] webUIQueue full, discarding signal BTtoMQTT           38:19:97
N: [ OMG->MQTT ] topic: home/OMG_ATOM_L/BTtoMQTT/582D34381997 msg: {"id":"58:2D:34:38:19:97","rssi":-90,"brand":"Xiaomi","model":"Mi Jia round","model_id":"LYWSDCGQ","type":"THB","tempc":29.6,"tempf":85.28,"hum":46,"mac":"58:2D:34:38:19:97"}

The above is output from the publishing Open Mqtt Gateway system, I am subscribing to the topic: home/OMG_ATOM_L/BTtoMQTT/# where the final topic element is the msg key (colon removed mac addy), message payloads may contain sensor data, partial sensor data, signal strength data, and battery status data , and they arrive to fast for the esp32 hosted console gui to keep up.

{"Id":"A4C1384647C3","Model":"TH Sensor","Mac":"A4:C1:38:46:47:C3"
,"TempC":27.3,"TempF":81.14,"Humidity":51,"Volt":2.722,"battery":56,"LastUpdated":1755037700755}

and

{"Id":"C47C8D6443C8","Model":"MiFlora","Mac":"C4:7C:8D:64:43:C8","Moisture":87,"TempC":27.9,"TempF":82.22,"LastUpdated":1755037696390,"Fertiliser":3979,"Lux":78,"battery":54}

are examples of the context objects currently held .

5

This is a slightly bigger/better example of what I said:

[
    {
        "id": "413e2398165b5905",
        "type": "mqtt out",
        "z": "a531e868a7ed264e",
        "name": "",
        "topic": "Device1",
        "qos": "",
        "retain": "",
        "respTopic": "",
        "contentType": "",
        "userProps": "",
        "correl": "",
        "expiry": "",
        "broker": "8941f4c3.0f151",
        "x": 1930,
        "y": 300,
        "wires": []
    },
    {
        "id": "f23b540f21caa8d4",
        "type": "inject",
        "z": "a531e868a7ed264e",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "",
        "payloadType": "date",
        "x": 1760,
        "y": 300,
        "wires": [
            [
                "413e2398165b5905"
            ]
        ]
    },
    {
        "id": "d1d6840752c246de",
        "type": "inject",
        "z": "a531e868a7ed264e",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "",
        "payloadType": "date",
        "x": 1760,
        "y": 420,
        "wires": [
            [
                "44c8feca63b2cedd"
            ]
        ]
    },
    {
        "id": "44c8feca63b2cedd",
        "type": "mqtt out",
        "z": "a531e868a7ed264e",
        "name": "",
        "topic": "Device2",
        "qos": "",
        "retain": "",
        "respTopic": "",
        "contentType": "",
        "userProps": "",
        "correl": "",
        "expiry": "",
        "broker": "8941f4c3.0f151",
        "x": 1930,
        "y": 420,
        "wires": []
    },
    {
        "id": "794309985d88880c",
        "type": "inject",
        "z": "a531e868a7ed264e",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "",
        "payloadType": "date",
        "x": 1760,
        "y": 540,
        "wires": [
            [
                "b07fcf1e997024e3"
            ]
        ]
    },
    {
        "id": "b07fcf1e997024e3",
        "type": "mqtt out",
        "z": "a531e868a7ed264e",
        "name": "",
        "topic": "Device3",
        "qos": "",
        "retain": "",
        "respTopic": "",
        "contentType": "",
        "userProps": "",
        "correl": "",
        "expiry": "",
        "broker": "8941f4c3.0f151",
        "x": 1930,
        "y": 540,
        "wires": []
    },
    {
        "id": "7285b46a64d90c2d",
        "type": "mqtt in",
        "z": "a531e868a7ed264e",
        "name": "",
        "topic": "BIRTH",
        "qos": "2",
        "datatype": "auto-detect",
        "broker": "8941f4c3.0f151",
        "nl": false,
        "rap": true,
        "rh": 0,
        "inputs": 0,
        "x": 2180,
        "y": 300,
        "wires": [
            [
                "7ae659a8c5755140"
            ]
        ]
    },
    {
        "id": "a9738cf2dcc38c34",
        "type": "mqtt in",
        "z": "a531e868a7ed264e",
        "name": "",
        "topic": "DEATH",
        "qos": "2",
        "datatype": "auto-detect",
        "broker": "8941f4c3.0f151",
        "nl": false,
        "rap": true,
        "rh": 0,
        "inputs": 0,
        "x": 2180,
        "y": 340,
        "wires": [
            [
                "7ae659a8c5755140"
            ]
        ]
    },
    {
        "id": "7ae659a8c5755140",
        "type": "switch",
        "z": "a531e868a7ed264e",
        "name": "who?",
        "property": "payload",
        "propertyType": "msg",
        "rules": [
            {
                "t": "cont",
                "v": "D1",
                "vt": "str"
            },
            {
                "t": "cont",
                "v": "D2",
                "vt": "str"
            },
            {
                "t": "cont",
                "v": "D3",
                "vt": "str"
            }
        ],
        "checkall": "true",
        "repair": false,
        "outputs": 3,
        "x": 2350,
        "y": 320,
        "wires": [
            [
                "fe15e8a3139d63a8"
            ],
            [
                "e73d5c37bb8e7b2f"
            ],
            [
                "b9e2bb854b641fc1"
            ]
        ]
    },
    {
        "id": "fe15e8a3139d63a8",
        "type": "change",
        "z": "a531e868a7ed264e",
        "name": "Device 1",
        "rules": [
            {
                "t": "set",
                "p": "Device1",
                "pt": "flow",
                "to": "topic",
                "tot": "msg"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 2520,
        "y": 270,
        "wires": [
            []
        ]
    },
    {
        "id": "e73d5c37bb8e7b2f",
        "type": "change",
        "z": "a531e868a7ed264e",
        "name": "Device 2",
        "rules": [
            {
                "t": "set",
                "p": "Device2",
                "pt": "flow",
                "to": "topic",
                "tot": "msg"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 2520,
        "y": 320,
        "wires": [
            []
        ]
    },
    {
        "id": "b9e2bb854b641fc1",
        "type": "change",
        "z": "a531e868a7ed264e",
        "name": "Device 3",
        "rules": [
            {
                "t": "set",
                "p": "Device3",
                "pt": "flow",
                "to": "topic",
                "tot": "msg"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 2520,
        "y": 370,
        "wires": [
            []
        ]
    },
    {
        "id": "2c39e0c814461fac",
        "type": "comment",
        "z": "a531e868a7ed264e",
        "name": "REMOTE DEVICES",
        "info": "",
        "x": 1870,
        "y": 260,
        "wires": []
    },
    {
        "id": "33c9171872589ad3",
        "type": "comment",
        "z": "a531e868a7ed264e",
        "name": "REMOTE DEVICES",
        "info": "",
        "x": 1870,
        "y": 380,
        "wires": []
    },
    {
        "id": "fcd21dfd7540173e",
        "type": "comment",
        "z": "a531e868a7ed264e",
        "name": "REMOTE DEVICES",
        "info": "",
        "x": 1870,
        "y": 500,
        "wires": []
    },
    {
        "id": "8941f4c3.0f151",
        "type": "mqtt-broker",
        "name": "Local",
        "broker": "127.0.0.1",
        "port": "1883",
        "clientid": "",
        "autoConnect": true,
        "usetls": false,
        "compatmode": false,
        "protocolVersion": 4,
        "keepalive": "60",
        "cleansession": true,
        "autoUnsubscribe": true,
        "birthTopic": "BIRTH",
        "birthQos": "0",
        "birthPayload": "D1 ONLINE",
        "birthMsg": {},
        "closeTopic": "",
        "closeQos": "0",
        "closePayload": "",
        "closeMsg": {},
        "willTopic": "DEATH",
        "willQos": "0",
        "willPayload": "D1 UNEXPECTED OFFLINE",
        "willMsg": {},
        "userProps": "",
        "sessionExpiry": ""
    }
]

But please understand the parts on the left are all different machines.
I can't show them exactly because they - in this case - are all on the same machine.
I hope you can see what I mean though.

6

Thanks for the input @Trying_to_learn I should be more clear.

I probably have made a design error, failed to adopt the principles of node red etc..

I have 16x environmental sensors for temp and humidity around the house used to input into heating control when required and many more plant pot sensors basically monitored to remind me to water..

Originally I used now depreciated nodes in an old version of node red hosted on an old os. in this system i would use the 8 yr old Xiaomi-ble-contrib node to connect to and query the sensors, its problem was it could take multiple minutes and retries to successfully connect and pull data making it a scheduling issue more than anything else.

I had to upgrade and in doing so discovered the end of dashboard 1 and that some dependencies had depreciated ..
Hence the use now of OMG to provide the mqtt input from the sensors without intervention :wink:

however if i have flows triggered/started by mqtt reciepts i will be checking if the heating needs to be on 20 times or more a minute a degree of granularity not required or desired.

So My bad design idea??? was to use the mqtt feed to update a context record of ‘current’ sensor readings.

Then use injection nodes to schedule flows as needed to log, display, decide.. keeping logic and gui traffic to a minimum.

I am a node red novice in that i played with node red to build version 1 it worked great and i left it alone for 6?years or so, now im back and its got all new features on top of everything i already forgot.

If using the context to hold a set of current sensor readings for reference etc is a bad idea its best i know now..

7

I don't think it is a BAD idea. Nor is using something that worked 6 years ago.

How DB2 works is different to the older one. But as long as you know the differences, it shouldn't be a problem.
You just have to work out if it is a one time fix or you want to rewrite the whole thing.
Neither is best. Just saying.

I can't see a problem with using context to hold the readings.
But it may be worth saving the context to the file system rather than memory.

Good luck with it.

8

I think that holding the readings in context is a reasonable idea.
You can choose to save the context to your SD card or hold it in memory.
Unless it's unacceptable to loose data if the Pi is rebooted or Node-red deployed, memory (the default) is probably best for you.

The selection of data you show is slightly worrying since it repeatedly says it's discarding messages.
Some message payloads are OK eg

image
image556×508 21.3 KB

while some seem to be corrupted, eg
image
image696×479 25.4 KB

I don't see msg.payload.key in these messages, are you using the MAC address, either with or without colons?

I don't know what the Open Mqtt Gateway system is?
Each device presumably publishes it's sensor data at intervals. Are the intervals fixed or do they just respond to a request?

Let's assume for now that all messages that arrive in Node-red are valid JSON.

I would probably use a function node to handle the incoming messages.

You seem to be looping through the properties of the message eg "id", "rssi" ... "batt", "volt", "mac" and updating that bit of the context store if it exists.
I can't immediately see the point of this. If it's a partial message with only some properties, you end up with some stale data in context.

Here is a very basic function to save the entire message into context using the mac address as the key, though it might not be the best choice.

image
image734×118 6.98 KB

let contextstore = flow.get('contextstore') ?? {}  // retrieve the data from context
const mac = msg.payload.mac   // take MAC address as the key
contextstore[mac] = msg.payload     // update 
flow.set('contextstore', contextstore)    // and save
// return msg; // If the function has no nodes wired downstream, no need to return

After importing the two messages above, cleaned up, the context store contains this. These two messages happen to have the same properties but if others differ, there is no problem, at least not when processing the input.

{
  "A4:C1:38:46:47:C3":{
    "id":"A4:C1:38:46:47:C3",
    "rssi":-70,
...
    "tempc":27.4,
    "tempf":81.32,
    "hum":51,
    "batt":54,
    "volt":2.71,
    "mac":"A4:C1:38:46:47:C3"
  },
  "58:2D:34:38:19:97":{
    "id":"58:2D:34:38:19:97",
...
    "tempc":29.6,
    "tempf":85.28,
    "hum":46,
    "mac":"58:2D:34:38:19:97"
  }
}
9

I tried to embed the flow in a message but its too big? so ive attached the json file if interested.

dont be too critical of the raw code especially the set up func (disabled) i am relearning..

You spotted my efficiency mod/edit @jbudd the real code addresses each required data element but a for each loop is easier to look at than a list of explicit if exists update statements especially in the convoluted form i pasted in..

the message ‘key’ is the colon removed mac addy and is pulled from the last element in the msg.topic not the payload.

My limited understanding is the advertising packets have a limited payload and not all sensor and status data can be encapsulated in a single packet. hence its broadcasted in a round robin with sparse data records..

the dropped messages refer to the gui console interface i believe? its hosted on an esp32. the actual publishing seems to be ample to keep everything updated ok.

OMG is a great project btw i discovered it when trying ble motorised radiator valves (3x aa batteries per radiator.. no thanks..)..

I dont have a problems as such beyond refining, and adding any control logic (the fun bit), its just i cant shake what seems like a mantra that function node are to be avoided..
btw the graphs are just placeholders to confirm i can pull data as required..

flows.json (31.0 KB)

10

Took a bit of time to digest the message, i see one of my errors now at least, i have been fixated on an array as the structure to contain my objects.. but an object structure will work much better creating a virtual sparse array much easier to index/access.

your code example is significantly more sophisticated than my step by step plod through.. i have a lot to digest still, thanks for the pointers..

I live in an urban environment close to busy amenities, sensors with a mac addy not set up or existing in the current list are expected to be ignored as they could be neighbours devices or in the street outside, thats why i used a set up function to sit listen and log devices for a short period and then i can edit any unwanted devices out before routing the incomming messages to the update function.

edit - just checked all my ‘valid’ sensors addresses start with 58, C4 or A4 currently, im only testing with 10x temp/humidity sensors and 6x plant pot sensors with batteries installed..

11

Had a coffee, and a walk around the kitchen and the fact you encapsulated the whole thing in 4 lines when it took me hours to debug my pseudo code into error free commands isnt a problem , not at all..

ok so as is it will add anything and overwrite a sparse record with another sparse record but neither are show stoppers and easily remedied, thanks again for the direction.

Im also beginning to see how to node-ify it i think.. a little more homework required reading up on join nodes

12

You pointed out that sometimes data is sent in a round robin arrangement, so my approach of replacing the entire stored data for that device is probably not appropriate, you will need to loop through the message properties somehow and handle each individually.

It's early morning here and I've not yet looked at your flow.

13

Oh its definitely rewrite time..

Thats the point… this end at least.

14

dont look now ive rewritten it, used 3x of your 4 lines and added 10x lines of message validation and exception message handling and another 10 lines of data element conditional assignments.

15

this is the current update func under test

let contextstore = flow.get('ObjList1') ?? {}  // retrieve the data from context
const mac = msg.payload.mac   // take MAC address as the key
var mymsg ={};
mymsg.topic = "ERR/Warning not an update record?";
mymsg.payload = msg.payload;
if(contextstore[mac] !== undefined){
    //test and assign data elements
    if (msg.payload.tempc !== undefined) contextstore[mac].tempc = msg.payload.tempc
    if (msg.payload.tempf !== undefined) contextstore[mac].tempf = msg.payload.tempf
    if (msg.payload.hum !== undefined) contextstore[mac].hum = msg.payload.hum
    if (msg.payload.batt !== undefined) contextstore[mac].batt = msg.payload.batt
    if (msg.payload.volt !== undefined) contextstore[mac].volt = msg.payload.volt
    if (msg.payload.moi !== undefined) contextstore[mac].moi = msg.payload.moi
    if (msg.payload.lux !== undefined) contextstore[mac].lux = msg.payload.lux
    if (msg.payload.fer !== undefined) contextstore[mac].fer = msg.payload.fer
    contextstore[mac].lastupdated = Date.now();
    mymsg.topic = "success obj updated";
    mymsg.payload = contextstore[mac];
}
flow.set('contextstore', contextstore);    // and save
msg = mymsg;
return msg;

I dropped the redundant id property, and adopted the lowercase property naming, and modified the set up func too accordingly -

Currently all exceptions are expected, dodgy message formats.. no mac property other required properties or not in know list..

I have a few tweaks to the rest of the nodes referencing previous property labels still..

next i need to consider a data store Time series db are new to me but not im not convinced that if the aggregate data is useful the underlying source data is disposable ?

Thanks again for the input and direction

16

You could optimize this like:

let contextstore = flow.get('ObjList1') ?? {};
const { mac, ...payload } = msg.payload;

let mymsg = { topic: "ERR/Warning not an update record?", payload: msg.payload };

if (contextstore[mac]) {
    // List of fields to update
    const fields = ["tempc", "tempf", "hum", "batt", "volt", "moi", "lux", "fer"];
    
    fields.forEach(field => {
        if (payload[field] !== undefined) contextstore[mac][field] = payload[field];
    });

    contextstore[mac].lastupdated = Date.now();

    mymsg = { topic: "success obj updated", payload: contextstore[mac] };
}

flow.set('ObjList1', contextstore); 
return mymsg;
17

In the above function you are taking the stored data from flow.ObjList1 but saving it back to flow.contextstore.

According to duckduckgo Search Assist, Javascript offers two ways to merge two objects which may have different properties (in your case the saved data and incoming message):
The spread operator ...

const obj1 = { a: 1, b: 2 };
const obj2 = { b: 3, c: 4 };
const mergedObj = { ...obj1, ...obj2 };
console.log(mergedObj); // Output: { a: 1, b: 3, c: 4 }

and Object.assign

const target = { a: 1, b: 2 };
const source = { b: 3, c: 4 };
const mergedObj = Object.assign(target, source);
console.log(mergedObj); // Output: { a: 1, b: 3, c: 4 }

This could replace your assignments to individual properties, but at the expense of having unnecessary extra properties in the context, eg id, rssi, etc.

18

Why not just rate limit the data as required and run with that. Why complicate it by using context?

19

Hi Colin, because on installing the shiney new version of node red i read about the context and thought hmmm, then when i adopted omg to repeat the ble broadcasts for me to filter out the relevant records and log/display/use them. i had a 2nd Hmm moment..

I dont know what i am doing, my first concern with posting was im using the context out of context excuse the attempt at a pun. I would happily accept (slightly disappointedly) im trying to hammer with a screwdriver to paraphrase.

While i didnt come here to get my homework done Im blown away with the suggestions above and am grateful for the direction.

20

My take on flow and global context is that they should only be used when it is of significant benefit. It is very easy to get race conditions using context. Almost always it is better to avoid it and do it the node-red way using messages.