Beginner questions on IF expressions and wildcard usage in function node

Colin · 3 September 2023 14:52

Does this work for you?

[{"id":"bfe082243e057b8b","type":"function","z":"bdd7be38.d3b55","name":"forEach()","func":"Object.keys(msg.payload).forEach((element) => {\n  if (element.startsWith(\"ATC\")) {\n    msg.payload = msg.payload[element]\n    node.send(msg)\n  }\n})\n","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":860,"y":160,"wires":[["68be1dc221018baa"]]},{"id":"68be1dc221018baa","type":"debug","z":"bdd7be38.d3b55","name":"debug 99","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1020,"y":160,"wires":[]},{"id":"3586ec0a6edd8630","type":"inject","z":"bdd7be38.d3b55","name":"No ATC","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"mac\":\"a4c1380d5c0c\",\"Temperature\":21.6,\"Humidity\":64.7,\"DewPoint\":14.7,\"Btn\":0,\"Battery\":37,\"RSSI\":-64}","payloadType":"json","x":690,"y":180,"wires":[["bfe082243e057b8b"]]},{"id":"3f53197d243846c3","type":"inject","z":"bdd7be38.d3b55","name":"With ATC","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"mac\":\"a4c1380d5c0c\",\"Temperature\":21.6,\"Humidity\":64.7,\"DewPoint\":14.7,\"Btn\":0,\"Battery\":37,\"ATC1\":\"this is the value of ATC1\",\"RSSI\":-64}","payloadType":"json","x":700,"y":140,"wires":[["bfe082243e057b8b"]]}]

Colin · 3 September 2023 14:54

Actually filter probably is better

[{"id":"68be1dc221018baa","type":"debug","z":"bdd7be38.d3b55","name":"debug 99","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1020,"y":160,"wires":[]},{"id":"3586ec0a6edd8630","type":"inject","z":"bdd7be38.d3b55","name":"No ATC","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"mac\":\"a4c1380d5c0c\",\"Temperature\":21.6,\"Humidity\":64.7,\"DewPoint\":14.7,\"Btn\":0,\"Battery\":37,\"RSSI\":-64}","payloadType":"json","x":690,"y":180,"wires":[["853adc83d89cddaa"]]},{"id":"853adc83d89cddaa","type":"function","z":"bdd7be38.d3b55","name":"filter()","func":"// get an array of all keys starting with ATC\nconst matchingKeys = Object.keys(msg.payload).filter((element) => element.startsWith(\"ATC\"))\n// We only expect one so matchingKeys[0] is the property we want, provided it exists\nif (matchingKeys.length > 0) {\n    msg.payload = msg.payload[matchingKeys[0]]\n    return msg\n}","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":870,"y":160,"wires":[["68be1dc221018baa"]]},{"id":"3f53197d243846c3","type":"inject","z":"bdd7be38.d3b55","name":"With ATC","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"mac\":\"a4c1380d5c0c\",\"Temperature\":21.6,\"Humidity\":64.7,\"DewPoint\":14.7,\"Btn\":0,\"Battery\":37,\"ATC1\":\"this is the value of ATC1\",\"RSSI\":-64}","payloadType":"json","x":700,"y":140,"wires":[["853adc83d89cddaa"]]}]

But either will work

TotallyInformation · 3 September 2023 14:55

Like I said, you need an Array as input to the filter function, not an object. If the id you want to filter on is contained in each entries data, you have to first filter on the "keys" - e.g. the property names of the object, then pass the filtered keys to a loop. If the id to filter on is contained in each entry then you can simply use the "values" and filter directly.

AleXSR700 · 3 September 2023 15:06

Thank you Colin

You write in your comments that we expect only one, so key will be [0]. This is not the case. As in the example I shared, there are often more than one.

With the current code only [0] is output. The second (and possibly more) would be discarded.

Is it a big change to output all in separate messages? So one msg.payload per matching element?

Colin · 3 September 2023 15:13

I thought you said there would only be one. Try this

[{"id":"bfe082243e057b8b","type":"function","z":"bdd7be38.d3b55","name":"forEach()","func":"Object.keys(msg.payload).forEach((element) => {\n  if (element.startsWith(\"ATC\")) {\n    node.send({payload: msg.payload[element]})\n  }\n})\n","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":860,"y":160,"wires":[["68be1dc221018baa"]]},{"id":"68be1dc221018baa","type":"debug","z":"bdd7be38.d3b55","name":"debug 99","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1020,"y":160,"wires":[]},{"id":"3586ec0a6edd8630","type":"inject","z":"bdd7be38.d3b55","name":"No ATC","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"mac\":\"a4c1380d5c0c\",\"Temperature\":21.6,\"Humidity\":64.7,\"DewPoint\":14.7,\"Btn\":0,\"Battery\":37,\"RSSI\":-64}","payloadType":"json","x":690,"y":180,"wires":[["bfe082243e057b8b"]]},{"id":"3f53197d243846c3","type":"inject","z":"bdd7be38.d3b55","name":"With ATC","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"mac\":\"a4c1380d5c0c\",\"Temperature\":21.6,\"Humidity\":64.7,\"DewPoint\":14.7,\"Btn\":0,\"Battery\":37,\"ATC1\":\"this is the value of ATC1\",\"RSSI\":-64}","payloadType":"json","x":700,"y":140,"wires":[["bfe082243e057b8b"]]},{"id":"aaaaf1995c5b6682","type":"inject","z":"bdd7be38.d3b55","name":"Multiple ATC*","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"mac\":\"a4c1380d5c0c\",\"Temperature\":21.6,\"Humidity\":64.7,\"DewPoint\":14.7,\"Btn\":0,\"Battery\":37,\"ATC1\":\"this is the value of ATC1\",\"ATC2\":\"this is the value of ATC2\",\"RSSI\":-64}","payloadType":"json","x":710,"y":220,"wires":[["bfe082243e057b8b"]]}]

Colin · 3 September 2023 15:18

Or using filter

[{"id":"3f53197d243846c3","type":"inject","z":"bdd7be38.d3b55","name":"With ATC","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"mac\":\"a4c1380d5c0c\",\"Temperature\":21.6,\"Humidity\":64.7,\"DewPoint\":14.7,\"Btn\":0,\"Battery\":37,\"ATC1\":\"this is the value of ATC1\",\"RSSI\":-64}","payloadType":"json","x":700,"y":140,"wires":[["853adc83d89cddaa"]]},{"id":"3586ec0a6edd8630","type":"inject","z":"bdd7be38.d3b55","name":"No ATC","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"mac\":\"a4c1380d5c0c\",\"Temperature\":21.6,\"Humidity\":64.7,\"DewPoint\":14.7,\"Btn\":0,\"Battery\":37,\"RSSI\":-64}","payloadType":"json","x":690,"y":180,"wires":[["853adc83d89cddaa"]]},{"id":"aaaaf1995c5b6682","type":"inject","z":"bdd7be38.d3b55","name":"Multiple ATC*","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"mac\":\"a4c1380d5c0c\",\"Temperature\":21.6,\"Humidity\":64.7,\"DewPoint\":14.7,\"Btn\":0,\"Battery\":37,\"ATC1\":\"this is the value of ATC1\",\"ATC2\":\"this is the value of ATC2\",\"RSSI\":-64}","payloadType":"json","x":710,"y":220,"wires":[["853adc83d89cddaa"]]},{"id":"853adc83d89cddaa","type":"function","z":"bdd7be38.d3b55","name":"filter()","func":"// get an array of all keys starting with ATC\nconst matchingKeys = Object.keys(msg.payload).filter((element) => element.startsWith(\"ATC\"))\n// send them\nmatchingKeys.forEach((element) => node.send({payload: msg.payload[element]}))","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":870,"y":200,"wires":[["68be1dc221018baa"]]},{"id":"68be1dc221018baa","type":"debug","z":"bdd7be38.d3b55","name":"debug 99","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1020,"y":160,"wires":[]}]

AleXSR700 · 3 September 2023 15:28

That is perfect!
Would it also be possible to use the pass the respective name ATC* as e.g. msg.topic?

I thought it would be as simple as doing

matchingKeys.forEach((element) => node.send({ topic: msg.payload.keys }))

but of course it was not

Colin · 3 September 2023 15:32

The key is in element so you can use
matchingKeys.forEach((element) => node.send({topic: element, payload: msg.payload[element]}))

AleXSR700 · 3 September 2023 15:37

The details. It's always the details that get me.

Thank you very much, this works perfectly!

Colin · 3 September 2023 16:56

Yes, computers are stupid, they do what you tell them instead of what you meant.

Please look at the code and make sure you understand how it works. If you don't understand anything then ask. Then the next time you come across a similar problem you will be more likely to be able to solve it yourself.

Nothing you are doing in that function is going to be significant compared to the other things that are going on with the message, just getting it in via MQTT and decoding it to a javascript object will be much more significant, plus whatever you are going to do with it later.

It is unusual to have such variability in the contents of MQTT messages, are they under your control? If so then perhaps a redesign of how the data are structured would make the rest of the system simpler. For example, if your ATC key value came through as part of the MQTT topic it might be better. Alternatively, rather than have
{"Time":"2023-09-03T14:48:50","ATC0d5c0c":{"mac":"a4c1380d5c0c","Temperature":21.6,"Humidity":66.2,"DewPoint":15,"Btn":0,"Battery":37,"RSSI":-68}
perhaps
{"Time":"2023-09-03T14:48:50","ATC":{"id": 0d5c0c", "mac":"a4c1380d5c0c","Temperature":21.6,"Humidity":66.2,"DewPoint":15,"Btn":0,"Battery":37,"RSSI":-68}
and make sure there is only one device per message. Or if you have to have more than one in each message
{"Time":"2023-09-03T14:48:50","ATC":[{"device": "0d5c0c", "mac":"a4c1380d5c0c","Temperature":21.6,"Humidity":66.2,"DewPoint":15,"Btn":0,"Battery":37,"RSSI":-68},{"device": "8f1b4f", "mac":"a4c1388f1b4f","Temperature":21.9,"Humidity":62.9,"DewPoint":14.5,"Btn":0,"Battery":69,"RSSI":-62},"TempUnit":"C"}]

AleXSR700 · 4 September 2023 18:58

Hello Colin,

yes, I am trying to understand as much as possible and then use the things I have learned to build off of them.

Unforunately not. The data structure is given by the respective source device. That can be anything from a Shelly running tasmota to a xiaomi mijia running pvvx's custom firmware.

The goal is to use Node-RED to process all of these different sources, change what is then output and format it in a more useful manner.

I am only at the start of my project (and my Node-RED learning journey)

EDIT:
I do have a question, where I think I know how it works but I am not yet sure how to make it work for me.
I found that if I want to only pass certain properties of a message I can do

        var msg = { payload: msg.payload.ENERGY['Power'] };
        return msg;

And if I want to pass the entire message and just change one property I can do this instead

    msg.topic="tele/"+(msg.topic.split("/")[1] || "").split("/")[0];
    msg.Time=msg.payload.Time
    return msg;

But what I have not figured out yet is how to use this in your node.send()

const matchingKeys = Object.keys(msg.payload).filter((element) => element.startsWith("ATC"))
matchingKeys.forEach((element) => node.send({ topic: msg.topic, payload: msg.payload[element] }))

So this only sends the topic and payload. But if I want to change one of the two and send the entire message, something like

node.send({ msg.payload=msg.payload[element] })

does not work.

EDIT:
Also tried

const matchingKeys = Object.keys(msg.payload).filter((element) => element.startsWith("ATC"))
matchingKeys.forEach((element) => { msg.payload=msg.payload[element]; return msg } )

unsuccessfully

Shouldn't everything behind =>just be a normal list of commands?

Colin · 4 September 2023 20:01

I misunderstood, I thought these were all coming in on the same MQTT topic. I see now the different cases are actually coming in on different topics from different devices.

What device is giving you the ATC data?

It might appear that you could just do

matchingKeys.forEach((element) => {
  msg.payload = msg.payload[element]
  node.send(msg)
})

But the problem is that the first time round the loop it would overwrite msg.payload and mess up the rest of the loop. The solution is to clone the message, modify it, and send that, so something like

matchingKeys.forEach((element) => {
    let newMsg = RED.util.cloneMessage(msg);
    newMsg.topic = element
    newMsg.payload = msg.payload[element]
    node.send(newMsg)
})

TotallyInformation · 4 September 2023 20:04

Though avoid this in a loop if you can because it is an expensive operation. If you don't need to worry about incoming msg properties other than topic and payload, simply create a new blank msg instead.

Colin · 4 September 2023 20:19

That is what the original solution did. The suggestion to clone it was specifically because of a new question - how to pass on the complete message, just modifying topic and payload. You are right though, cloning the message will add significantly to the processing. Whether this matters is unknown though.

smcgann99 · 5 September 2023 00:29

@AleXSR700 I started this learning process a few years ago with my first foray into home automation.

As the system grew organically over time my knowledge improved with lots of help from this forum .

I like to learn by doing it on the fly and hate planning things, however every now and again I revisit early code that has been working fine, because I find it limits current enhancements. I often laugh at my old code, thing what newbie wrote this

I would suggest you take pause and look at the big picture, how you deal with the incoming data is probably the most important thing to get right. You will want to do things with it in the future that you haven't even thought of as yet!

So with regard to this specific thread, I seems you may be using wild cards on your incoming topics to try and capture and deal with every type of message in one flow ?

It may be better to at least limit each flow to deal with a specific type of device, Where the message structure is well documented and consistent.

eg one for Tasmota and another for shelly. It will be easier to process as there are less variations in messages to deal with. You could then craft the differing payloads into your own "standard" and pass this format to use along the flows, and it will not matter what type of device it originated from.

Something I wish I had done at the beginning

Just my two cents, anyway good luck

AleXSR700 · 5 September 2023 05:37

Thank you, Colin.
My understanding of the function was not too far off then actually, but I would not have thought of the fact that the payload is being changed for all future processing.

I thought that the incoming message would be cached and remain unaltered, and all processing is applied to the outgoing message. So each iteration would start with the cached original.
Clearly this is not the case.

You are right. I did not consider this. I was trying to keep as much of the original message as possible (also linked to what smcgann99 says) because I want to keep my options open for further processing.

But for performance purposes I will keep my original approach on this particular subflow

I completely agree with you. That is why I am taking my time building this new structure.
I originally treated the data at source, which was difficult because tasmota and other IoT interfaces are limited in their programming capabilities due to flash size. So I hit a wall of what can be done and the awesome people in the tasmota support recommended I go the next step and process elsewhere.

Like you I was thinking of separating the data early and then keeping my options open as much as possible.
That is why I am also trying to keep the original data as long as possible.
Here a quick overview of my early stage of this project:

[Btw: is it possible to format text in a label bold/italic in the name field? I found that \n allows line breaks but \b did not work for bold.]

As you can see, I am first separating per device type (based on the syntax the incoming messages have). More device specific subflows will follow in the next few days.
I then separate into the different kinds of information that I am interested in.
In the upper "ATC" flow I will have to replace the original message as a whole to keep processing fast. But that means I need to change the earlier nodes everytime I forget something.

On the lower flow "Energy" I am keeping the original msg and I only change certain parts or use a filter to "route"/"split" the data for processing.

Big picture:
The idea is to receive all mqtt data from all my sensors on a dedicated topic that Home Assistant is not subscribed to and then filter out information (data I do not care about like wifi signal strength or firmware version) and also perform a redundancy check by comparing previous values to the current one and then only forwarding data that has changed significantly enough to be forwarded (if anybody here uses tasmota: kind of like powerdelta).

Given the number of messages coming in (~50 devices sending updates every few seconds), I need to keep the processing as simple and fast as possible. I am sure my code will evolve a lot over time

It is great to know that there are dedicated people on this forum who are willing (and capable!) to help newbies like myself

jbudd · 5 September 2023 07:28

No criticism intended here, just interested observations:

My first thought looking at that flow diagram was that you have an awful lot of function nodes.
On reflection, each of your functions is probably relatively simple, so no doubt more easily maintained than a single complex function.

I do wonder why you have no switch nodes to route messages along one wire or another based on the message properties, nor change nodes to adjust the message structure for your requirements.

It looks like your output may be multiple messages, each with a single item of data.
I wonder why you need this?
For example, a dashboard gauge can as easily display the value of msg.payload.something[0].temperature (or from the same message, msg.payload.something[0].dewpoint) as it can msg.temperature.

It would be interesting to see the whole flow exported, if you are happy to share it.

Colin · 5 September 2023 07:59

msg is just a javascript object. If the code modifies it then it is modified.

To concur with @jbudd's comments, it appears you are bringing all the data in from different devices under the same topic and then examining the data to work out what is in it. That is horribly inefficient. For example you will be checking all the properties of messages to see if they are ATC*, when you know that they are actually from a different device type. Either have a Switch node after the MQTT node, testing the topic, and splitting the message down different paths dependent on device type, or have multiple MQTT In nodes, one for each device type. I don't think there is a significant difference in efficiency between those two approaches.

AleXSR700 · 5 September 2023 10:49

I do have other flows where I use the "simpler" switch, change etc. nodes. But I have found that albeit being super simple in the beginning, they quickly become bottle necks.
So I decided to try to do as much as possible with functions. This has the added benefit of being able to learn much more and perform more complex tasks in the future

The debug nodes are actually not my real output. They are really just for debugging right now. The output will later be an mqtt out node.
So in the current stage of my little project, I am separating the data for processing.
This will be followed by a block of functions that will process the data.
Once processed, the different streams will be re-joined to form one output message (per device... see more on that in my response to Colin)

Perfectly valid points

Maybe I need to elaborate a bit more.

Device types are different hardware types (usually linked to different manufacturers).
So I have:

Shelly 1PM, Shelly 2.5 and Shelly 3EM.
Gosund SP1 and Gosund SP112.
Refoss P11
Xiaomi Mijia Thermometer & Hygrometer
eq3 Thermostat (heating valve)
Devices are then "one piece of hardware" no matter who the manufacturer. Entities are then e.g. power sensor, energy total sensor, power switch etc.

The first three device types are running with tasmota firmware. So they will all have the respective device_id in the topic, e.g. "tele/tasmota_12345678" which is based on the MAC of the devices.
These run to the bottom flow because they are all power sensors and their messages are all very similar.
So the first function checks if it is one of these device types. Since the topic contains the device id, I do not need to split into devices anymore. The second function then splits into the entities Energy Power, Energy Total and Energy Today.
However, those with multiple outputs (e.g. Shelly 2.5 or Shelly 3EM) will need separate processing of these 3 outputs, hence then another function splitting into the entities Power[0] to Power[3].

The Xiaomi and eq3 are in fact bluetooth devices with their own firmware (whose behavior I cannot really influence). They are relayed through an ESP32 dev kit board running tasmota.
So the data comes in via bluetooth to the ESP32 and the ESP32 then relays/publishes the data as MQTT messages via Wifi.
Each Xiaomi device uses "ATC" + a unique ID based on MAC to identify itself to tasmota.

Tasmota does not use the same topic as it does for the Shelly/Gosund device types. This alternative topic is not device specific but device type specific. Meaning, all Xiaomi Mijia devices' data are published via one large message with just one topic (they may be split into multiple messages if there are more than 4 Xiaomi devices).

So I need the top flow to check the message for "ATC" and not the topic.
The same applies to the eq3 valves, which I have not implemented yet because they are not used in summer (and are essentially clone nodes of the ATC flow).

Maybe as an additional note:
I could of course join different functions into one and have less processing steps. But right now this makes it easier to work out mistakes and to follow the processing.
IF this leads to performance problems later, I will merge them into fewer but larger functions.
Until then I think it is easier for you to support and me to learn if I keep each step in a separate node.

P.S.:
Of course I am happy to share the final glow later. It is currently in a private github repo but once I have a more refined and more feature rich flow, I will happily share it with whoever is interested.

Colin · 5 September 2023 11:19

Which is the first function? The MQTT node feeds two functions.