How can I filter out 1 JSON input from another?

[questuk]

Hi,

I am receiving msg.payload.humidity and msg.payload.temperature correctly on debug payload node.

You can see i have setup a change node. the debug 54 gets the correct msg.payload.temperature, but I guess also the msg.payload.humidity causing a undefined.

How do I stop the unwanted msg.payload.humidity also being sent to the debug 54 node ?

A short GIF ...

Thanks

[Steve-Mcl]

Put a switch node (before the debug) with "has key" and test msg.payload has key temperature

[questuk]

Thanks again Steve,

That solved that part.

The other reason I asked the question was that I have Humidity & Temperature gauges setup.

Now as shown both gauges work as expected, but the reason I added the debug was to see what was actually going to the Temperature gauge and I saw undefined as well as the correct value.

The change payload.temperature node, does not block all messages as shown when I used the debug node ?

So basically I am now asking, am I doing this correctly for the gauges ?

Thanks

[Steve-Mcl]

@questuk

The change payload.temperature node, does not block all messages as shown when I used the debug node ?

No, the change node is for changing things. The switch node is for switching (AKA routing) messages.

@questuk

So basically I am now asking, am I doing this correctly for the gauges ?

Thanks

Impossible to tell as you dont show what is coming in via the link-in node (i.e. if you are sending an object with both humidity AND temperature then this is fine. If you are still sending separate objects (1 with temperature and 1 with humidity`) then no! you will need a switch before the change node to prevent the msg passing if it hasn't got the data you want in it.

[questuk]

Hi Steve

Thanks for being so patient

In a previous question on here you recommended to ..

@Steve-Mcl

PS:

a better topic structure would be module/22/xxxxx this way you can subscribe to all modules module/# or a single module module/22/# or a specif topic of all modules module/+/temperature

So that is what I have done, the topic structure is now ... module/99/data/output/json

My intention is to output all data from my Arduino sensors using this but just changing the 99 to an Arduino micro controller ID in my home, there are about 10 of them all with there own ID ... 99 , 10, 11 12, etc.

Then selecting the data I require using the topic and the correct ID, with the various payloads I require ...
payload.temperature
payload.humidity
payload.lighting
payload.PIR
etc
Example ... module/12/data/output/json then I require separately temperature and lighting from this Arduino.

So am I doing the extraction of the data correctly or is there a more simple way ?

As at the moment it seem as I have to use ...

Hopefully that all make sense
It's just as I am changing how I was doing it, to the new way I want to get it correct from the start.
Thanks again

[Colin]

You don't need the Change nodes that move msg.payload.temperature (or the others) into msg.payload. In the Gauge node you can set the Value Format field to {{msg.payload.temperature}} and it will automatically pick up the temperature from the payload.

If you changed your topics so that the temperature value (just the value as a number) was published to module/12/data/temperature (or similar) then you would not need the switch nodes. You could configure one MQTT node for each parameter and feed it directly into the gauge.

[questuk]

@Colin

You don't need the Change nodes that move msg.payload.temperature (or the others) into msg.payload. In the Gauge node you can set the Value Format field to {{msg.payload.temperature}} and it will automatically pick up the temperature from the payload.

Hi Colin yes I remember you saying this last time and I did try this and yes it does work.
But I was using the above as an example as I have a lot of information coming and going from my Node-RED. Lighting, database, heating, daylight and dusk etc. and as I am moving to using JSON I wanted to make sure I am not doing something wrong.

If you changed your topics so that the temperature value (just the value as a number) was published to module/12/data/temperature (or similar) then you would not need the switch nodes. You could configure one MQTT node for each parameter and feed it directly into the gauge.

More or less this is what I have been using for the last 5 years and it does get very confusing. JSON seemed easier as I could just select msg.payload.temperature or msg.payload.daylight etc.

My database is setup in Node-RED to read & write JSON so it seems the way to go for the rest of my flows etc.

Any thoughts or recomendations are welcome

[Colin]

@questuk

More or less this is what I have been using for the last 5 years and it does get very confusing.

Why is having the topic module/12/data/output/json containing sometimes {temperature: 27} and sometimes {humidity: 75} less confusing than topic module/12/data/temperature containing 27 and module/12/data/humidity containing 75?

[questuk]

HI Colin,

I do understand what you mean, but it is all because of my Arduino code. I will try to explain.
I have multiple Arduino modules around my house, the Arduino code is multi functional.

So all of them run the same code, BUT with just a change of a define, I can change how that module monitors various sensors.

Example
Module 1 ... measures humidity and temperature and daylight and PIR.
Module 2 ... measures daylight and if dark it operates lighting.
Module 3 .. Receives my spoken message from Alexa and this tells me the set temperature and actual temperature of a room.

All doing very different things but just one master Arduino sketch to select what I want them to do.

All did have there own MQTT output message, then I would have to add that to NR on multiple tabs.

I am hoping with JSON Objects and LINK in and OUT nodes to simplify by just monitoring module/XX/data/output/json etc and then using the power of Node-RED to select the messages I require.

This is a project that has evolved over 7 or 8 years and with little knowledge, somethings I did right and some I could definitely improve on.

Hard to explain but it gives you an idea and what I am trying to achieve.

Thanks

[shrickus]

From your debug output, it seems that every msg has a single reading, under different property names. As Colin suggests, every sensor should use it's own topic, if only for clarification while debugging. You can still have single mqtt in node with a wildcard topic module/# that receives all the sensor readings. And you can still pass along a JSON payload with a named property (i.e. msg.payload.humidity) just like you are showing, if you want to.

The goal is to have every dashboard widget only receive the messages it can show -- which is why Steve suggested you use a switch node to route the messages, and why Colin likes separate mqtt nodes, to route them properly. Seems like the same number of "routing" nodes are needed in either case, equal to the number of measurements you want to display.

A simpler way I've used is to use the wildcard mqtt node to receive all measurements, and then use a function or change node to "merge" the 1 new reading into a flow context "object" which holds ALL the measurements that have been received. Then on a separate flow you can use an inject node set to send the context object periodically, which updates each widget on the dashboard. As Colin suggested, each gauge or meter just displays a single property from the same context object -- so no msg routing is needed.

The logic that stores each incoming reading will depend upon your topic/message structure. For instance, if your msg.topic was "module/12/temperature" and your JSON msg.payload included was { "temperature": 27 }, you could use this JS code in a function to merge it with your flow context object (untested):

let readings = flow.get("AllReadings") || {};  // get any existing flow readings
let {prefix, module, type} = msg.topic.split("/");  // split the topic into module and type
let reading = readings[module] || {};  // get or init the new reading for this module
reading[type] = msg.payload[type];  // set the new reading value for this module
flow.set("AllReadings", readings); // save all readings back into flow context

(I'm sure someone will point out any errors in this code ...)

The equivalent way of doing that is to use a change node to set flow.AllReadings to the output of this JSONata expression:

(
    $parts := topic.$split("/");
    $readings := $flowContext("AllReadings") ? $flowContext("AllReadings") : {};
    $readings[$parts[1]] := $merge($readings[$parts[1]], payload);
    $readings
)

[questuk]

shrickus This was sent as you were typing your resonse

Hi,

Doing my research online and found this and it seems to work.
What do you think ?

[shrickus]

Seems we cross-posted, and my assumption that each msg contains just one reading is not true? In that case, you can ignore my comments about including the type of reading in the topic. But I would still put the module number in the topic like you are doing -- and the technique of merging new messages into a context object still works using this modified expression:

(
    $parts := topic.$split("/");
    $readings := $flowContext("AllReadings") ? $flowContext("AllReadings") : {};
    $readings[$parts[1]] := $merge($readings[$parts[1]], payload);
    $readings
)

Please note: much of this code does not do adequate error checking... It's just a quick way to show the concept.

Yes, that change expression does in fact move the msg.payload.temperature value into msg.payload, but you can do the same things without adding the overhead of invoking JSONata:

[questuk]

Thanks for all your help guys,

I appreciate all you time trying to help me.
Shrickus .. I have very limited knowledge of coding and my brain works better using visual cues,

So I have gone with Steves idea, and I didn't realise I can have 2 outputs on the switch node, so less clutter on my Node-RED flows. I can also expand this for any other messages I require by adding another output .. etc.

And Colin, I will actually put the JSON object value into the Gauge, as you mentioned ( but like I said I used this on here as an example )

Thanks all

[jbudd]

@questuk

So that is what I have done, the topic structure is now ... module/99/data/output/json

Then selecting the data I require using the topic and the correct ID, with the various payloads I require

Sorry but I'm not convinced you have done it in the best way

Looking at your original diagram:

It seems that your arduino is sending two messages

{
"payload": {temperature: 22}
}

and

{
"payload": {"humidity": 29}
}

When the second, humidity message arrives, your attempt to move msg.payload.temperature to msg.payload results in undefined.

I think the expectation was that you send a single message with both properties:

{
"payload": {"temperature": 22, "humidity": 29}
}

[questuk]

Hi, Jbudd,

Thanks ...

As I am testing my system at the moments and making gradual changes to my Arduino code.
Yes at the moment I am sending it as 2 separate messages.

The other day I was running some tests and it confirmed it does work correctly for temperature and humidity when both sent in one JSON object.

Today I was adjusting my Master Arduino code and at the moment it has 2 separate pieces of code for both. ( This is how it has been for many years )

But I will most probably change it soon to output one Object and I may even send out everything in one go as some of my Arduino's runs on batteries and sleep a lot, so they may benefit from one quick blast of data then sleep again.

But I always treat a problem as a learning experience, sorry to all on here, I do need your help from time to time, but I do try hard to try and find the answer on the internet before asking.

Thanks