Extracting part of the data from JSON payload, perform calcs and combine them back

General
1

Hi Guys,

finally i have my device (based on esp01) up and running, reporting some environmental data via MQTT (mosquito) to Node-RED. The msg.payload contains an object with BME280 sensor readings, PZEM-004T readings together with 32 integers from multiplexed ADC channels - these are NTC measurements (temperature probes). The problem i have is i need to use Steinhart&Hart equation to get the temperature values out of ADC integers then format this whole bunch of information and send to influxDB as a single measurement point. In addition i'm using this approach:

meaning i'm doing kind of bridge between mqtt payload received from esp02 and influxDB so i know the database will not be messed up ifsome transmission bugs occur from esp01 and i can use shorter keys names (so smaller buffer in esp01 for mqtt).
What is working:

  1. receiving data from esp01
  2. Steinhart&Hart calculation done in function node as Java Script (for one channel so far)
  3. formatting msg.payload for influxDB so all the field and tags are nicely named and send to database.

Where i stuck:

  1. so far i'm sending raw integers values (measurements from ADC) to database. I know how to extract particular ADC measurement and calculate temperature, but i don't know how to do it for all 32 channels in one script (probably fro loop to be used) and then combine these 32 temerature values (floats) with the message to be send to database.

Hope the description is clear enough:)
thanks!

2

Seems like you've done all the hard parts and you are stumbling over the easy part!

We need to know what format you have your input data in if you want some help though. You are correct that you almost certainly need a loop in your function node assuming all the data is in one msg.

Incidentally, I use a similar process for writing to InfluxDB now. Where any output goes via a specific flow that makes sure all of the data is in the correct format before output. Makes life a lot easier, especially because the format for InfluxDB output is something I'm always forgetting.

3

Thanks for reply, i appreciate it.

Incoming JSON is like this (from the top of my head i don't have an access to my system now):

msp.payload = 
{"device":"logger","temp":22.4,"press":101325,"hum":43.1,"rssi":51.3,"reconnections":4233,"voltage":234.2,"current":1.2,"freq":60.0,"power":281,"pf"=0.99,"energy":1232,"gas":10000,"ADC":[13,124,21432,342432,123,32434,5342,23,23,5,43,63,6346,24,51,6,1,46,6346,34,6,1,654,6,5,345,2353,23236,74573,3532,64642,64564]}

As you can see all data flying from my esp01 device is in one packet via MQTT to Node-RED. My flow first uses JSON node to convert it into object. For now JSON node output goes to DEBUG node, CHANGE node and FUNCTION nodes simultanously. CHANGE node is formatiing the output for influxDB like this:

[{"temperature":22.4,"pressure":101325,"humidity":43.1,"voltage":234.2,"current":1.2,"frequency":60.0,"power":281,"power_factor"=0.99,"energy":1232,"gas-level":10000},{"device_ID":"heater logger","location":"heating_room"}]

FUNCTION node for now is taking just one value from ADC values array (in the example channel 1) and calculating NTC temperature using Steinhart-Hart quation:

var R = Number(msg.payload.ADC[1];
var R0 = 10000;
var T0 = 273.15 + 25;
var B = 3950;
var T = 1 / ( (1/T0) + (1/B) * Math.log(R/R0) );
T = T - 273.15;
msg.payload = T;
return msg;

and then it's just connected to DEBUG node.

Now i'd like to do this:

  1. load ADC array elements and 'analyze' each value as there will be the following cases:
    a). value below certain treshold - NTC probe is shorted or switch connected instead of probe
    b). value above certain treshold - NTC probe disconnected or switch connected instead of probe
    c). NTC healthy and i measuring the temperature

FUNCTION node is to be creating two arrays as output:
1st: NTC calculated temperatures values (floats) array
2nd: Alternative functions array, let's call it now MODE, containing integers related to each incoming ADC channels with value as follows (example):
- value "1" for case a). - lever below treshold
- value "0" for case b). - level above treshild
- value "-1" for case c) so NTC probe connected and healthy

This is so i can later in my flow check if certain switch is on or off if connected instead of NTC (the switch can reflect i.e. circulating pump status running/stopped). If value is "-1" then the flow will now there is now switch and do nothing as there is NTC on this channel so another part of the flow will calculate temperature and push it to influxDB.

So the final task is to combine the above mentioned array (or at least the one with calculated temperatures) with msg.payload from CHANGE node to create the ultimate data set for influxDB measurement point.

Complicated?:slight_smile:

4

Not really but without your flow and sample data it is difficult to advise.

If you could capture a real data packet (use the copy value button on a debug message), put that in an inject to simulate data & paste a minimal flow in your reply I will take a look for you.

5

Thank you Steve. Good morning.

i don't fully inderstand this:

put copied debug content (meaning JSON object i'm sending from esp9266) and put that in an inject (inject node?) to simulate data (how is it?) & paste minimal flow (?). I'm sorry for my stupid questions but i'm at the beginning of my learning curve in Node-Red and IoT...

6

If I asked you to post your flow as it is, I would see what you have done but I would NOT have any data to test it (because I dont have your esp9266 device!)

So if you grab a sample of data (using the copy value button) and paste that into an inject - so that is simulates data coming from your esp9266 device - then you can post a copy of your flow (export flow using CTRL+E) that I can use to try and assist you.


For reference...

There’s a great page in the docs that will explain how to use the debug panel

Pay particular attention to the part about the buttons that appear under your mouse pointer when you over hover a debug message property in the sidebar.

BX00Cy7yHi

https://nodered.org/docs/user-guide/messages

7

Brilliant. let me get back home after work so i can do it. Meanwhile i'll go through the link and some docs to gain knowledge on the suubject!:slight_smile: Thank you.
Actually the incoming json string will be like the one i've already written in my second post. As you need the flow i will have to do a separate one as in the flow i have many other things - most of them is my trial&error playing with different things, including receiving communicates from another esp8266's and sending messages to yet another esp8266's. So if the "copy flow" feature will copy whole flow it will be huge and messy. Let me isolate nodes that are in use for the topic in question.
Thank you for you time Steve.

8

Somehow i managed to do it:) Here's my flow:

[{"id":"cba5893f.b5c308","type":"tab","label":"Flow 1","disabled":false,"info":""},{"id":"a9f190af.e7aa9","type":"debug","z":"cba5893f.b5c308","name":"","active":false,"tosidebar":true,"console":false,"tostatus":true,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":450,"y":220,"wires":[]},{"id":"653e64eb.dbccbc","type":"json","z":"cba5893f.b5c308","name":"","property":"payload","action":"","pretty":false,"x":430,"y":300,"wires":[["5516c553.6963d4","7baa4d8d.637a0c","35f8ae62.633cea"]]},{"id":"5516c553.6963d4","type":"debug","z":"cba5893f.b5c308","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":570,"y":360,"wires":[]},{"id":"68610877.529038","type":"influxdb out","z":"cba5893f.b5c308","influxdb":"c567e10e.6094b8","name":"kanal1","measurement":"NTC1","precision":"","retentionPolicy":"","database":"database","precisionV18FluxV20":"ms","retentionPolicyV18Flux":"","org":"organisation","bucket":"bucket","x":850,"y":280,"wires":[]},{"id":"35f8ae62.633cea","type":"function","z":"cba5893f.b5c308","name":"oblicz temperature NTC","func":"//skrypt testowy, do kanału ADC[1] podłączony jest termistor NTC\n//10000 ohm o współczynniku B=3950. \nvar R = Number(msg.payload.ADC[1]);\nvar R0 = 10000;\nvar T0 = 273.15 + 25;\nvar B = 3950;\nvar T =  1 / ( (1/T0) + (1/B) * Math.log(R/R0) );\nT = T - 273.15;\nmsg.payload = T;\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":630,"y":280,"wires":[["311283c.16bb87c","68610877.529038"]]},{"id":"311283c.16bb87c","type":"debug","z":"cba5893f.b5c308","name":"","active":true,"tosidebar":true,"console":false,"tostatus":true,"complete":"payload","targetType":"msg","statusVal":"payload","statusType":"auto","x":750,"y":220,"wires":[]},{"id":"7baa4d8d.637a0c","type":"change","z":"cba5893f.b5c308","name":"przygotowanie danych do influxDB","rules":[{"t":"set","p":"payload","pt":"msg","to":"[\t   {\t       \"cisnienie\" : msg.payload.pressure,\t       \"temperatura\" : msg.payload.temperature,\t       \"wilgotnosc\" : msg.payload.humidity,\t       \"napieceie_sieciowe\" : msg.payload.voltage,\t       \"pobor_pradu\" : msg.payload.current,\t       \"moc_pobierana\" : msg.payload.power,\t       \"licznik_energii\" : msg.payload.energy,\t       \"czestotliwosc\" : msg.payload.frequency,\t       \"PF\" : msg.payload.power_factor,\t       \"czujnik_gazu\" : msg.payload.methane,\t       \"kanal_1\" : msg.payload.ADC[0],\t       \"kanal_2\" : msg.payload.ADC[1],\t       \"kanal_3\" : msg.payload.ADC[2],\t       \"kanal_4\" : msg.payload.ADC[3],\t       \"kanal_5\" : msg.payload.ADC[4],\t       \"kanal_6\" : msg.payload.ADC[5],\t       \"kanal_7\" : msg.payload.ADC[6],\t       \"kanal_8\" : msg.payload.ADC[7],\t       \"kanal_9\" : msg.payload.ADC[8],\t       \"kanal_10\" : msg.payload.ADC[9],\t       \"kanal_12\" : msg.payload.ADC[10],\t       \"kanal_13\" : msg.payload.ADC[11],\t       \"kanal_14\" : msg.payload.ADC[12],\t       \"kanal_15\" : msg.payload.ADC[13],\t       \"kanal_16\" : msg.payload.ADC[14],\t       \"kanal_17\" : msg.payload.ADC[15],\t       \"kanal_18\" : msg.payload.ADC[16],\t       \"kanal_19\" : msg.payload.ADC[17],\t       \"kanal_20\" : msg.payload.ADC[18],\t       \"kanal_21\" : msg.payload.ADC[19],\t       \"kanal_22\" : msg.payload.ADC[21],\t       \"kanal_23\" : msg.payload.ADC[22],\t       \"kanal_24\" : msg.payload.ADC[23],\t       \"kanal_25\" : msg.payload.ADC[24],\t       \"kanal_26\" : msg.payload.ADC[25],\t       \"kanal_27\" : msg.payload.ADC[26],\t       \"kanal_28\" : msg.payload.ADC[27],\t       \"kanal_29\" : msg.payload.ADC[28],\t       \"kanal_30\" : msg.payload.ADC[29],\t       \"kanal_31\" : msg.payload.ADC[30],\t       \"kanal_32\" : msg.payload.ADC[31]\t    },\t   {\t       \"ID_urzadzenia\" : msg.payload.device,\t       \"lokalizacja\": \"kotlownia\"\t    }\t]","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":660,"y":320,"wires":[["52703314.59cdfc","da36f675.4888e8"]]},{"id":"52703314.59cdfc","type":"influxdb out","z":"cba5893f.b5c308","influxdb":"c567e10e.6094b8","name":"szymmmm","measurement":"opomiarowanie_kotlowni","precision":"","retentionPolicy":"","database":"database","precisionV18FluxV20":"ms","retentionPolicyV18Flux":"","org":"organisation","bucket":"bucket","x":930,"y":320,"wires":[]},{"id":"da36f675.4888e8","type":"debug","z":"cba5893f.b5c308","name":"","active":true,"tosidebar":true,"console":false,"tostatus":true,"complete":"payload","targetType":"msg","statusVal":"payload","statusType":"auto","x":910,"y":360,"wires":[]},{"id":"ad25f82c.79f5b8","type":"inject","z":"cba5893f.b5c308","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"device\":\"esp01-Sz\",\"reconnections\":6910,\"RSSI[dB]\":-50,\"pressure\":101126,\"temperature\":22.02,\"humidity\":53.18,\"voltage\":235.5,\"current\":0,\"power\":0.5,\"energy\":0.196,\"frequency\":60,\"power_factor\":1,\"methane\":10000,\"ADC\":[6.305,11438.816,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,996.465,-1,1476.66,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1]}","payloadType":"str","x":250,"y":260,"wires":[["a9f190af.e7aa9","653e64eb.dbccbc"]]},{"id":"c567e10e.6094b8","type":"influxdb","hostname":"influxdb","port":"8086","protocol":"http","database":"pierwsza_baza","name":"","usetls":false,"tls":"","influxdbVersion":"1.x","url":"http://localhost:8086","rejectUnauthorized":true,"info":"\"pierwsza_baza\" to baza testowa, którą stworzyłem do pierwszych prób. "}]
9

I don't have my Node-RED on hand now but i did a draft for FUNCTION node that will be taking data (msg.payload) from JSON node, replacing original values in ADC array with calculated and this node will be connected to CHANGE node which will do formatting for influxDB. This is the script for FUNCTION node:

//replace ADC reading with calculated temperature values
var i
for (i = 0; i < msg.payload.ADC.lenght; i++){
var R = Number(msg.payload.ADC[i];
if (R < 0){msg.payload.ADC[i] = null} //let's put null if particular ADC reading is -1 (esp01 is sending -1 if ADC input is shorted). Does null mean there will be nothing in array at 'i' index?
else{ //calculate temperature
var R0 = 10000;
var T0 = 273.15 + 25;
var B = 3950;
var T = 1 / ( (1/T0) + (1/B) * Math.log(R/R0) );
T = T - 273.15;
msg.payload.ADC[i] = T; //and store it in appropriate place in ADC array - overwrite original value
}
}

//now it's time to send everything to CHANGE node for formatting for influxDB
return msg.payload;

10

Guys,
the flow works, i'm receiving data from ESP01, parsing as Java Script Object, using Function node to make calculations, then Change node to format data for InfluxDB and sending them to influxdb. The problem I can't figure out now is not all 32 teperature channels are reported each time. The result now is the channel which is off is reporting some value wwhich is not true and should not be written info InfluxDB. In my Function node, where i'm testing if the value is valid and if it should be calculated as temperature value to be written to database, i'm putting "null" if value is no correct and should not be written. Unfortunatelly it makes the next node, Change node (where i'm formatting data for InfluxDB), reporting error, saying "wrong data format". It looks like "null" is not valid type for JSON? I was trying to find some info on how to use IF...THEN-a-like method for JSONata but can't make it work. Please, any hints how to overcome this issue? Thank you.

11

it is not necessary to write null. You can just not include the sensors in the data going to influx.

Edit: So for example instead of
{sensor1: 3, sensor2: 54, sensor3: 7}
if sensor2 is not available then use
{sensor1: 3, sensor3: 7}

12

Thanks Colin. The problem is that i never know when data is not valid so i need to use the method that will automatically include valid data and omitt invalid ones.
If one temperature probe (NTC) will go down it will block everything now - no data from other 31 probes will be saved and displayed on the trend. If i loose BME280 sensor (happen already) - it will make all other values blocked as well and so forth. My goal is just to bypass particular data if it turned out unavailable.

13

Try this function

let device = msg.payload.device;
msg.payload = {
       "cisnienie" : msg.payload.pressure,
       "temperatura" : msg.payload.temperature,
       "wilgotnosc" : msg.payload.humidity,
       "napieceie_sieciowe" : msg.payload.voltage,
       "pobor_pradu" : msg.payload.current,
       "moc_pobierana" : msg.payload.power,
       "licznik_energii" : msg.payload.energy,
       "czestotliwosc" : msg.payload.frequency,
       "PF" : msg.payload.power_factor,
       "czujnik_gazu" : msg.payload.methane,
       "ADC": msg.payload.ADC}

for (let i = 0; i < msg.payload.ADC.length; i++){
    let R = Number(msg.payload.ADC[i]);
    if (R >= 0){
        msg.payload[`kanal${(i+1)}`] = (1 / ( (1/298.15) + (1/3950) * Math.log(R/1000) ))-273.15;
    }
}
delete msg.payload.ADC;
msg.payload = [msg.payload, {"ID_urzadzenia" : device, "lokalizacja": "kotlownia"}];
return msg;
14

Thanks E1cid,
it looks like your approach will eliminate Change node completely. Am i correct? I'm trying to understand it. I can see i can do any conditional check on incoming data and thus reject unwanted/invalid data.
Haven't seen this construction before:
msg.payload[kanal${(i+1)}] = .......

15

yes that is correct, didn't see the need for the two.

`kanal${(i+1)}`

this is the same as

"kanal" + (i+1)
16

Perfect. I'll try it today after work. Thank you:)
Update: it worked! I got rid of change node and did as you suggested. Now, in Grafana, i need to fight newly arised issue: interpolation lines between data points. I've read, in Grafana, there's an option "Null values" that can be set to "disconnected" under graph options -> "Stacking and null values". Hope my RB Pi based Grafana version supports this option. Thanks again!:slight_smile:

17

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.