Joining weather data from 2 locations

[Paul-Reed]

The above flow obtains the current weather conditions from 2 different locations using Darksky.
The Parser function nodes parse out the apparent temp & cloud conditions, and this is then joined using a join node. The output from the join node looks like;

The db Format function node puts the data into a suitable format to pass onto a influxdb node and write the data to a database.

All good so far, everything works as expected. BUT I want to avoid the join node getting out of sync if one of the Darksky api call fails, so I've set a 60 second timeout in the join node, but then I get an error because the db Format function node because the 2nd weather location data is missing.

The db Format function node contains;

var cala = msg.payload.calagaldana;
var sac = msg.payload.sacoma;

msg.payload = [{
    CG_apptemp: cala.CG_apptemp,
    CG_cloud: cala.CG_cloud,
    SC_apptemp: sac.SC_apptemp,
    SC_cloud: sac.SC_cloud
},
{
    metrics: "weather"
}];

msg.measurement = "conditions";
return msg;

I've tried a few things like reworking the array as;

CG_apptemp: msg.payload.calagaldana.CG_apptemp || null, 

Which really didn't work out!
Any suggestion how best to mitigate the error please.

[moebius]

Only one parser needed...the result should look like this and works for as many locations you want:
{
apptemp: apptemp,
cloud: cloud,
location: "calagaldana"
}

The join node should create an array which you iterate in the db node...
Easier way: the db could be normalized and you just send the weather data without joining...

[Paul-Reed]

@moebius

Only one parser needed...the result should look like this and works for as many locations you want:

I don't see how that would improve error handling. The flow works fine when both Darksky nodes return data, the error arises when the api for one of the locations fails.

In that case, I really would like to see something like;
0: object
CG_apptemp: null
CG_cloud: null
SC_apptemp: 23.65
SC_cloud: 0
being fed into the influxdb node.

Edit - not possible. Cannot post null values to influx

@moebius

the db could be normalized and you just send the weather data without joining

Until tonight, I did send the data without joining, but it makes it more difficult to analyse in Grafana, as the data points would not be aligned when hovering over.

[moebius]

ic...sorry, dont know how to do that

[dceejay]

The join node won’t know how to get initialised with nulls so I would think building your own join with a function node would be easiest.

[Paul-Reed]

@dceejay

building your own join with a function node would be easiest.

Hmm, that's going to be fun

[dceejay]

Ok - then - maybe the formatter can insert the null data if needs be ?

CG_apptemp: cala.CG_apptemp || null;

etc

(of course that simple version will break if temp is exactly 0

[Paul-Reed]

@dceejay

(of course that simple version will break if temp is exactly 0

Yep, that's what I found when I tried that earlier. The cloud cover was 0, which was interpreted as a null.

@Paul-Reed

I've tried a few things like reworking the array as;

CG_apptemp: msg.payload.calagaldana.CG_apptemp || null, 

Which really didn't work out!

I'm also wondering if the influxdb node can handle a null value anyway, as when it tried to send the data containing a null, I got a 400 bad request error from the influxdb node.

That being the case, maybe instead of sending;

CG_apptemp: 30.17
CG_cloud: 0
SC_apptemp: null
SC_cloud: null

I should be just sending;

CG_apptemp: 30.17
CG_cloud: 0

In fact - just read a git issue response by @mike in which he confirms this;

...as I understand it, influxdb 1.0 and later does not support sending a point with null values. If you want to indicate that a value is missing, one suggested technique is that you use another field to indicate missing data.

[shrickus]

Paul,

I would use a change node after your join node -- using a JSONata expression to build the influxdb input data structure by merging whatever objects it receives:

[
	$merge(payload),
	{
    	"metrics": "weather"
	}
]

This will only work if each object in your payload array has unique key names (CG_apptemp vs. SC_apptemp) -- so that looks to be fine for this use case.

The nice thing about JSONata expressions is that the output silently drops any null objects. If there is a chance that some of the data values will be null, we can also process them out, but the $merge() function will not do that for you automatically I don't believe.

[Paul-Reed]

@shrickus

The nice thing about JSONata expressions is that the output silently drops any null objects.

It doesn't seem to drop the null values, see the brief flow below;

[{"id":"532e211d.7cdb9","type":"function","z":"c53060.842a0fa","name":"test data","func":"msg.payload = {\n    CG_apptemp: 28,\n    CG_cloud: 0,\n    SC_apptemp: null,\n    SC_cloud: null\n};\n\nreturn msg;\n","outputs":1,"noerr":0,"x":410,"y":1120,"wires":[["4a8794d2.3c8aec"]]},{"id":"7eb46d1.98d7494","type":"debug","z":"c53060.842a0fa","name":"","active":true,"console":"false","complete":"false","x":740,"y":1120,"wires":[]},{"id":"4a8794d2.3c8aec","type":"change","z":"c53060.842a0fa","name":"merge data","rules":[{"t":"set","p":"payload","pt":"msg","to":"[\t\t$merge(payload),\t\t{\t    \t\"metrics\": \"weather\"\t\t}\t]","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":570,"y":1120,"wires":[["7eb46d1.98d7494"]]},{"id":"13605a8b.89aab5","type":"inject","z":"c53060.842a0fa","name":"","topic":"","payload":"","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":"","x":260,"y":1120,"wires":[["532e211d.7cdb9"]]}]

Also, as the join node creates a key/value object;

it does not get processed by the JSONata merge.

The complete flow with the JSONata merge node added is;

[{"id":"c5ba0375.379","type":"darksky","z":"c53060.842a0fa","darksky":"","name":"Cala Galdana","lon":"3.960460","lat":"39.938713","date":"","time":"","mode":"node","lang":"en","units":"uk2","x":308,"y":840,"wires":[["d1c36626.4042e8"]]},{"id":"73043ed0.c3f35","type":"inject","z":"c53060.842a0fa","name":"2pm every day","topic":"","payload":"","payloadType":"str","repeat":"","crontab":"0 0-23 * * *","once":false,"onceDelay":"","x":120,"y":840,"wires":[["c5ba0375.379","80b98bb1.b56a28"]]},{"id":"e2505289.f076a","type":"darksky","z":"c53060.842a0fa","darksky":"","name":"Sa Coma","lon":"3.376063","lat":"39.579944","date":"","time":"","mode":"node","lang":"en","units":"uk2","x":330,"y":890,"wires":[["13cf9dfe.00fb62"]]},{"id":"a69c3819.a82f28","type":"join","z":"c53060.842a0fa","name":"","mode":"custom","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"60","count":"2","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":588,"y":840,"wires":[["a24646b6.b7a5c8","4b53014d.4a7c6"]]},{"id":"d1c36626.4042e8","type":"function","z":"c53060.842a0fa","name":"Parser","func":"msg.topic = \"calagaldana\";\nconst i = msg.data.currently;\n    const at = i.apparentTemperature;\n    const cl = i.cloudCover*100;\n\nmsg.payload = {\n    CG_apptemp: at,\n    CG_cloud: cl\n};\nreturn msg;\n","outputs":1,"noerr":0,"x":458,"y":840,"wires":[["a69c3819.a82f28"]]},{"id":"13cf9dfe.00fb62","type":"function","z":"c53060.842a0fa","name":"Parser","func":"msg.topic =\"sacoma\";\nconst i = msg.data.currently;\n    const at = i.apparentTemperature;\n    const cl = i.cloudCover*100;\n\nmsg.payload = {\n    SC_apptemp: at,\n    SC_cloud: cl\n};\nreturn msg;\n","outputs":1,"noerr":0,"x":460,"y":890,"wires":[["a69c3819.a82f28"]]},{"id":"80b98bb1.b56a28","type":"delay","z":"c53060.842a0fa","name":"","pauseType":"delay","timeout":"15","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"x":182,"y":890,"wires":[["e2505289.f076a"]]},{"id":"66dfa018.9ce1","type":"debug","z":"c53060.842a0fa","name":"","active":true,"console":"false","complete":"false","x":900,"y":840,"wires":[]},{"id":"a24646b6.b7a5c8","type":"change","z":"c53060.842a0fa","name":"merge data","rules":[{"t":"set","p":"payload","pt":"msg","to":"[\t\t$merge(payload),\t\t{\t    \t\"metrics\": \"weather\"\t\t}\t]","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":740,"y":840,"wires":[["66dfa018.9ce1"]]},{"id":"4b53014d.4a7c6","type":"debug","z":"c53060.842a0fa","name":"","active":true,"console":"false","complete":"false","x":740,"y":890,"wires":[]}]

[shrickus]

@Paul-Reed

The nice thing about JSONata expressions is that the output silently drops any null objects.

It doesn't seem to drop the null values, see the brief flow below;

Right, that why I said it drops null objects, not null values... and your incoming payload does not have any null objects ;*)

Anyway, I did miss that fact that the incoming payload was a single payload object -- my bad!
So to merge the objects stored in the payload properties you need a slightly different syntax, like so:

[
	$merge(payload.*),
	{
    	"metrics": "weather"
	}
]

[Paul-Reed]

Steve, that's a great solution, and works well
If either of the weather api's do not respond, the database is now only updated with the working api result, which is exactly what I wanted.
Thanks for the help.

Paul

[Paul-Reed]

If it helps anyone else in the future, here is the flow;

[{"id":"c5ba0375.379","type":"darksky","z":"c53060.842a0fa","darksky":"","name":"Cala Galdana","lon":"3.960460","lat":"39.938713","date":"","time":"","mode":"node","lang":"en","units":"uk2","x":308,"y":840,"wires":[["d1c36626.4042e8"]]},{"id":"73043ed0.c3f35","type":"inject","z":"c53060.842a0fa","name":"2pm every day","topic":"","payload":"","payloadType":"str","repeat":"","crontab":"0 0-23 * * *","once":false,"onceDelay":"","x":120,"y":840,"wires":[["c5ba0375.379","80b98bb1.b56a28"]]},{"id":"e2505289.f076a","type":"darksky","z":"c53060.842a0fa","darksky":"","name":"Sa Coma","lon":"3.376063","lat":"39.579944","date":"","time":"","mode":"node","lang":"en","units":"uk2","x":330,"y":890,"wires":[["13cf9dfe.00fb62"]]},{"id":"a69c3819.a82f28","type":"join","z":"c53060.842a0fa","name":"","mode":"custom","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"30","count":"2","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":588,"y":840,"wires":[["a24646b6.b7a5c8"]]},{"id":"d1c36626.4042e8","type":"function","z":"c53060.842a0fa","name":"Parser","func":"msg.topic = \"calagaldana\";\nconst i = msg.data.currently;\n    const at = i.apparentTemperature;\n    const cl = i.cloudCover*100;\n\nmsg.payload = {\n    CG_apptemp: at,\n    CG_cloud: cl\n};\nreturn msg;\n","outputs":1,"noerr":0,"x":458,"y":840,"wires":[["a69c3819.a82f28"]]},{"id":"13cf9dfe.00fb62","type":"function","z":"c53060.842a0fa","name":"Parser","func":"msg.topic =\"sacoma\";\nconst i = msg.data.currently;\n    const at = i.apparentTemperature;\n    const cl = i.cloudCover*100;\n\nmsg.payload = {\n    SC_apptemp: at,\n    SC_cloud: cl\n};\nreturn msg;\n","outputs":1,"noerr":0,"x":460,"y":890,"wires":[["a69c3819.a82f28"]]},{"id":"80b98bb1.b56a28","type":"delay","z":"c53060.842a0fa","name":"","pauseType":"delay","timeout":"10","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"x":182,"y":890,"wires":[["e2505289.f076a"]]},{"id":"a24646b6.b7a5c8","type":"change","z":"c53060.842a0fa","name":"merge data","rules":[{"t":"set","p":"payload","pt":"msg","to":"[\t\t$merge(payload.*),\t\t{\t    \t\"metrics\": \"weather\"\t\t}\t]","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":740,"y":840,"wires":[[]]}]