Tried filter() approach in the message above yours. Not successful yet 
Regarding the if expression: Thank you for the clear explanation. Maybe that is why it is so fast. It jumps directly and then fails rather than iterating through the array.
Please post the full function using my method.
In fact, having tried using filter, I think it gives a more complex solution.
Even iterating the properties will only take milliseconds, if that.
Your code was the function. I did not add to it.
So the payload I posted "went into your code" and the output was then either the payload of ATC* or 'undefined'.
So I think it was not dismissing the non-matches.
But I agree that filter() is the approach to continue with 
... if I can get it to work.
P.S.: Milliseconds may still be too long. There will a large number of messages coming in which will each need processing. So efficiency will become relevant.
Does this work for you?
[{"id":"bfe082243e057b8b","type":"function","z":"bdd7be38.d3b55","name":"forEach()","func":"Object.keys(msg.payload).forEach((element) => {\n if (element.startsWith(\"ATC\")) {\n msg.payload = msg.payload[element]\n node.send(msg)\n }\n})\n","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":860,"y":160,"wires":[["68be1dc221018baa"]]},{"id":"68be1dc221018baa","type":"debug","z":"bdd7be38.d3b55","name":"debug 99","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1020,"y":160,"wires":[]},{"id":"3586ec0a6edd8630","type":"inject","z":"bdd7be38.d3b55","name":"No ATC","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"mac\":\"a4c1380d5c0c\",\"Temperature\":21.6,\"Humidity\":64.7,\"DewPoint\":14.7,\"Btn\":0,\"Battery\":37,\"RSSI\":-64}","payloadType":"json","x":690,"y":180,"wires":[["bfe082243e057b8b"]]},{"id":"3f53197d243846c3","type":"inject","z":"bdd7be38.d3b55","name":"With ATC","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"mac\":\"a4c1380d5c0c\",\"Temperature\":21.6,\"Humidity\":64.7,\"DewPoint\":14.7,\"Btn\":0,\"Battery\":37,\"ATC1\":\"this is the value of ATC1\",\"RSSI\":-64}","payloadType":"json","x":700,"y":140,"wires":[["bfe082243e057b8b"]]}]
Actually filter probably is better
[{"id":"68be1dc221018baa","type":"debug","z":"bdd7be38.d3b55","name":"debug 99","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1020,"y":160,"wires":[]},{"id":"3586ec0a6edd8630","type":"inject","z":"bdd7be38.d3b55","name":"No ATC","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"mac\":\"a4c1380d5c0c\",\"Temperature\":21.6,\"Humidity\":64.7,\"DewPoint\":14.7,\"Btn\":0,\"Battery\":37,\"RSSI\":-64}","payloadType":"json","x":690,"y":180,"wires":[["853adc83d89cddaa"]]},{"id":"853adc83d89cddaa","type":"function","z":"bdd7be38.d3b55","name":"filter()","func":"// get an array of all keys starting with ATC\nconst matchingKeys = Object.keys(msg.payload).filter((element) => element.startsWith(\"ATC\"))\n// We only expect one so matchingKeys[0] is the property we want, provided it exists\nif (matchingKeys.length > 0) {\n msg.payload = msg.payload[matchingKeys[0]]\n return msg\n}","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":870,"y":160,"wires":[["68be1dc221018baa"]]},{"id":"3f53197d243846c3","type":"inject","z":"bdd7be38.d3b55","name":"With ATC","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"mac\":\"a4c1380d5c0c\",\"Temperature\":21.6,\"Humidity\":64.7,\"DewPoint\":14.7,\"Btn\":0,\"Battery\":37,\"ATC1\":\"this is the value of ATC1\",\"RSSI\":-64}","payloadType":"json","x":700,"y":140,"wires":[["853adc83d89cddaa"]]}]
But either will work
1 Like
Like I said, you need an Array as input to the filter function, not an object. If the id you want to filter on is contained in each entries data, you have to first filter on the "keys" - e.g. the property names of the object, then pass the filtered keys to a loop. If the id to filter on is contained in each entry then you can simply use the "values" and filter directly.
1 Like
Thank you Colin
You write in your comments that we expect only one, so key will be [0]. This is not the case. As in the example I shared, there are often more than one.
With the current code only [0] is output. The second (and possibly more) would be discarded.
Is it a big change to output all in separate messages? So one msg.payload per matching element?
I thought you said there would only be one. Try this
[{"id":"bfe082243e057b8b","type":"function","z":"bdd7be38.d3b55","name":"forEach()","func":"Object.keys(msg.payload).forEach((element) => {\n if (element.startsWith(\"ATC\")) {\n node.send({payload: msg.payload[element]})\n }\n})\n","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":860,"y":160,"wires":[["68be1dc221018baa"]]},{"id":"68be1dc221018baa","type":"debug","z":"bdd7be38.d3b55","name":"debug 99","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1020,"y":160,"wires":[]},{"id":"3586ec0a6edd8630","type":"inject","z":"bdd7be38.d3b55","name":"No ATC","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"mac\":\"a4c1380d5c0c\",\"Temperature\":21.6,\"Humidity\":64.7,\"DewPoint\":14.7,\"Btn\":0,\"Battery\":37,\"RSSI\":-64}","payloadType":"json","x":690,"y":180,"wires":[["bfe082243e057b8b"]]},{"id":"3f53197d243846c3","type":"inject","z":"bdd7be38.d3b55","name":"With ATC","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"mac\":\"a4c1380d5c0c\",\"Temperature\":21.6,\"Humidity\":64.7,\"DewPoint\":14.7,\"Btn\":0,\"Battery\":37,\"ATC1\":\"this is the value of ATC1\",\"RSSI\":-64}","payloadType":"json","x":700,"y":140,"wires":[["bfe082243e057b8b"]]},{"id":"aaaaf1995c5b6682","type":"inject","z":"bdd7be38.d3b55","name":"Multiple ATC*","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"mac\":\"a4c1380d5c0c\",\"Temperature\":21.6,\"Humidity\":64.7,\"DewPoint\":14.7,\"Btn\":0,\"Battery\":37,\"ATC1\":\"this is the value of ATC1\",\"ATC2\":\"this is the value of ATC2\",\"RSSI\":-64}","payloadType":"json","x":710,"y":220,"wires":[["bfe082243e057b8b"]]}]
Or using filter
[{"id":"3f53197d243846c3","type":"inject","z":"bdd7be38.d3b55","name":"With ATC","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"mac\":\"a4c1380d5c0c\",\"Temperature\":21.6,\"Humidity\":64.7,\"DewPoint\":14.7,\"Btn\":0,\"Battery\":37,\"ATC1\":\"this is the value of ATC1\",\"RSSI\":-64}","payloadType":"json","x":700,"y":140,"wires":[["853adc83d89cddaa"]]},{"id":"3586ec0a6edd8630","type":"inject","z":"bdd7be38.d3b55","name":"No ATC","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"mac\":\"a4c1380d5c0c\",\"Temperature\":21.6,\"Humidity\":64.7,\"DewPoint\":14.7,\"Btn\":0,\"Battery\":37,\"RSSI\":-64}","payloadType":"json","x":690,"y":180,"wires":[["853adc83d89cddaa"]]},{"id":"aaaaf1995c5b6682","type":"inject","z":"bdd7be38.d3b55","name":"Multiple ATC*","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"mac\":\"a4c1380d5c0c\",\"Temperature\":21.6,\"Humidity\":64.7,\"DewPoint\":14.7,\"Btn\":0,\"Battery\":37,\"ATC1\":\"this is the value of ATC1\",\"ATC2\":\"this is the value of ATC2\",\"RSSI\":-64}","payloadType":"json","x":710,"y":220,"wires":[["853adc83d89cddaa"]]},{"id":"853adc83d89cddaa","type":"function","z":"bdd7be38.d3b55","name":"filter()","func":"// get an array of all keys starting with ATC\nconst matchingKeys = Object.keys(msg.payload).filter((element) => element.startsWith(\"ATC\"))\n// send them\nmatchingKeys.forEach((element) => node.send({payload: msg.payload[element]}))","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":870,"y":200,"wires":[["68be1dc221018baa"]]},{"id":"68be1dc221018baa","type":"debug","z":"bdd7be38.d3b55","name":"debug 99","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1020,"y":160,"wires":[]}]
1 Like
That is perfect!
Would it also be possible to use the pass the respective name ATC* as e.g. msg.topic?
I thought it would be as simple as doing
matchingKeys.forEach((element) => node.send({ topic: msg.payload.keys }))
but of course it was not 
The key is in element so you can use
matchingKeys.forEach((element) => node.send({topic: element, payload: msg.payload[element]}))
1 Like
The details. It's always the details that get me.
Thank you very much, this works perfectly!
Yes, computers are stupid, they do what you tell them instead of what you meant.
Please look at the code and make sure you understand how it works. If you don't understand anything then ask. Then the next time you come across a similar problem you will be more likely to be able to solve it yourself.
Nothing you are doing in that function is going to be significant compared to the other things that are going on with the message, just getting it in via MQTT and decoding it to a javascript object will be much more significant, plus whatever you are going to do with it later.
It is unusual to have such variability in the contents of MQTT messages, are they under your control? If so then perhaps a redesign of how the data are structured would make the rest of the system simpler. For example, if your ATC key value came through as part of the MQTT topic it might be better. Alternatively, rather than have
{"Time":"2023-09-03T14:48:50","ATC0d5c0c":{"mac":"a4c1380d5c0c","Temperature":21.6,"Humidity":66.2,"DewPoint":15,"Btn":0,"Battery":37,"RSSI":-68}
perhaps
{"Time":"2023-09-03T14:48:50","ATC":{"id": 0d5c0c", "mac":"a4c1380d5c0c","Temperature":21.6,"Humidity":66.2,"DewPoint":15,"Btn":0,"Battery":37,"RSSI":-68}
and make sure there is only one device per message. Or if you have to have more than one in each message
{"Time":"2023-09-03T14:48:50","ATC":[{"device": "0d5c0c", "mac":"a4c1380d5c0c","Temperature":21.6,"Humidity":66.2,"DewPoint":15,"Btn":0,"Battery":37,"RSSI":-68},{"device": "8f1b4f", "mac":"a4c1388f1b4f","Temperature":21.9,"Humidity":62.9,"DewPoint":14.5,"Btn":0,"Battery":69,"RSSI":-62},"TempUnit":"C"}]
Hello Colin,
yes, I am trying to understand as much as possible and then use the things I have learned to build off of them.
Unforunately not. The data structure is given by the respective source device. That can be anything from a Shelly running tasmota to a xiaomi mijia running pvvx's custom firmware.
The goal is to use Node-RED to process all of these different sources, change what is then output and format it in a more useful manner.
I am only at the start of my project (and my Node-RED learning journey)
EDIT:
I do have a question, where I think I know how it works but I am not yet sure how to make it work for me.
I found that if I want to only pass certain properties of a message I can do
var msg = { payload: msg.payload.ENERGY['Power'] };
return msg;
And if I want to pass the entire message and just change one property I can do this instead
msg.topic="tele/"+(msg.topic.split("/")[1] || "").split("/")[0];
msg.Time=msg.payload.Time
return msg;
But what I have not figured out yet is how to use this in your node.send()
const matchingKeys = Object.keys(msg.payload).filter((element) => element.startsWith("ATC"))
matchingKeys.forEach((element) => node.send({ topic: msg.topic, payload: msg.payload[element] }))
So this only sends the topic and payload. But if I want to change one of the two and send the entire message, something like
node.send({ msg.payload=msg.payload[element] })
does not work.
EDIT:
Also tried
const matchingKeys = Object.keys(msg.payload).filter((element) => element.startsWith("ATC"))
matchingKeys.forEach((element) => { msg.payload=msg.payload[element]; return msg } )
unsuccessfully
Shouldn't everything behind =>just be a normal list of commands?
I misunderstood, I thought these were all coming in on the same MQTT topic. I see now the different cases are actually coming in on different topics from different devices.
What device is giving you the ATC data?
It might appear that you could just do
matchingKeys.forEach((element) => {
msg.payload = msg.payload[element]
node.send(msg)
})
But the problem is that the first time round the loop it would overwrite msg.payload and mess up the rest of the loop. The solution is to clone the message, modify it, and send that, so something like
matchingKeys.forEach((element) => {
let newMsg = RED.util.cloneMessage(msg);
newMsg.topic = element
newMsg.payload = msg.payload[element]
node.send(newMsg)
})
Though avoid this in a loop if you can because it is an expensive operation. If you don't need to worry about incoming msg properties other than topic and payload, simply create a new blank msg instead.
That is what the original solution did. The suggestion to clone it was specifically because of a new question - how to pass on the complete message, just modifying topic and payload. You are right though, cloning the message will add significantly to the processing. Whether this matters is unknown though.
1 Like
@AleXSR700 I started this learning process a few years ago with my first foray into home automation.
As the system grew organically over time my knowledge improved with lots of help from this forum 
.
I like to learn by doing it on the fly and hate planning things, however every now and again I revisit early code that has been working fine, because I find it limits current enhancements. I often laugh at my old code, thing what newbie wrote this 
I would suggest you take pause and look at the big picture, how you deal with the incoming data is probably the most important thing to get right. You will want to do things with it in the future that you haven't even thought of as yet!
So with regard to this specific thread, I seems you may be using wild cards on your incoming topics to try and capture and deal with every type of message in one flow ?
It may be better to at least limit each flow to deal with a specific type of device, Where the message structure is well documented and consistent.
eg one for Tasmota and another for shelly. It will be easier to process as there are less variations in messages to deal with. You could then craft the differing payloads into your own "standard" and pass this format to use along the flows, and it will not matter what type of device it originated from.
Something I wish I had done at the beginning 
Just my two cents, anyway good luck 
1 Like
Thank you, Colin.
My understanding of the function was not too far off then actually, but I would not have thought of the fact that the payload is being changed for all future processing.
I thought that the incoming message would be cached and remain unaltered, and all processing is applied to the outgoing message. So each iteration would start with the cached original.
Clearly this is not the case.
You are right. I did not consider this. I was trying to keep as much of the original message as possible (also linked to what smcgann99 says) because I want to keep my options open for further processing.
But for performance purposes I will keep my original approach on this particular subflow
I completely agree with you. That is why I am taking my time building this new structure.
I originally treated the data at source, which was difficult because tasmota and other IoT interfaces are limited in their programming capabilities due to flash size. So I hit a wall of what can be done and the awesome people in the tasmota support recommended I go the next step and process elsewhere.
Like you I was thinking of separating the data early and then keeping my options open as much as possible.
That is why I am also trying to keep the original data as long as possible.
Here a quick overview of my early stage of this project:
[Btw: is it possible to format text in a label bold/italic in the name field? I found that \n allows line breaks but \b did not work for bold.]
As you can see, I am first separating per device type (based on the syntax the incoming messages have). More device specific subflows will follow in the next few days.
I then separate into the different kinds of information that I am interested in.
In the upper "ATC" flow I will have to replace the original message as a whole to keep processing fast. But that means I need to change the earlier nodes everytime I forget something.
On the lower flow "Energy" I am keeping the original msg and I only change certain parts or use a filter to "route"/"split" the data for processing.
Big picture:
The idea is to receive all mqtt data from all my sensors on a dedicated topic that Home Assistant is not subscribed to and then filter out information (data I do not care about like wifi signal strength or firmware version) and also perform a redundancy check by comparing previous values to the current one and then only forwarding data that has changed significantly enough to be forwarded (if anybody here uses tasmota: kind of like powerdelta).
Given the number of messages coming in (~50 devices sending updates every few seconds), I need to keep the processing as simple and fast as possible. I am sure my code will evolve a lot over time
It is great to know that there are dedicated people on this forum who are willing (and capable!) to help newbies like myself 
1 Like
