# Joining weather data from 2 locations

**URL:** <https://discourse.nodered.org/t/joining-weather-data-from-2-locations/12589>\
**Category:** General\
**Created:** [25 June 2019 18:55 UTC](https://discourse.nodered.org/t/joining-weather-data-from-2-locations/12589 "2019-06-25T18:55:38Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Paul-Reed](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/paul-reed/32/66906_2.png) [@Paul-Reed](https://discourse.nodered.org/u/Paul-Reed)\
**Post date:** [25 June 2019 18:55 UTC](https://discourse.nodered.org/t/joining-weather-data-from-2-locations/12589/1 "2019-06-25T18:55:38Z")

</div>

![holweather](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/2X/0/026be08f488705e5c47dc2f56ff5fb8dd70afa98.jpeg)  
The above flow obtains the current weather conditions from 2 different locations using Darksky.  
The `Parser` function nodes parse out the apparent temp & cloud conditions, and this is then joined using a join node. The output from the join node looks like;

![1](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/2X/c/c570d0b6e9801fc9d0dfe7a9ecab1f5c715c9dab.jpeg)

The `db Format` function node puts the data into a suitable format to pass onto a influxdb node and write the data to a database.

All good so far, everything works as expected. BUT I want to avoid the join node getting out of sync if one of the `Darksky` api call fails, so I've set a 60 second timeout in the join node, but then I get an error because the `db Format` function node because the 2nd weather location data is missing.

![error](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/2X/b/b3b1987ad6a9ae656be9f7c89e1e486257bf2956.jpeg)

The `db Format` function node contains;

```auto
var cala = msg.payload.calagaldana;
var sac = msg.payload.sacoma;

msg.payload = [{
    CG_apptemp: cala.CG_apptemp,
    CG_cloud: cala.CG_cloud,
    SC_apptemp: sac.SC_apptemp,
    SC_cloud: sac.SC_cloud
},
{
    metrics: "weather"
}];

msg.measurement = "conditions";
return msg;

```

I've tried a few things like reworking the array as;

```auto
CG_apptemp: msg.payload.calagaldana.CG_apptemp || null, 

```

Which really didn't work out!  
Any suggestion how best to mitigate the error please.

---

<div class="post-metadata">

**Author:** ![moebius](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/moebius/32/2872_2.png) [@moebius](https://discourse.nodered.org/u/moebius)\
**Post date:** [25 June 2019 19:14 UTC](https://discourse.nodered.org/t/joining-weather-data-from-2-locations/12589/2 "2019-06-25T19:14:07Z")

</div>

Only one parser needed...the result should look like this and works for as many locations you want:  
{  
apptemp: apptemp,  
cloud: cloud,  
location: "calagaldana"  
}

The join node should create an array which you iterate in the db node...  
Easier way: the db could be normalized and you just send the weather data without joining...

---

<div class="post-metadata">

**Author:** ![Paul-Reed](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/paul-reed/32/66906_2.png) [@Paul-Reed](https://discourse.nodered.org/u/Paul-Reed)\
**Post date:** [25 June 2019 20:30 UTC](https://discourse.nodered.org/t/joining-weather-data-from-2-locations/12589/3 "2019-06-25T20:30:20Z")

</div>

> [@moebius](#):
>
> Only one parser needed...the result should look like this and works for as many locations you want:

I don't see how that would improve error handling. The flow works fine when both Darksky nodes return data, the error arises when the api for one of the locations fails.

~~In that case, I really would like to see something like;  
0: object  
CG\_apptemp: null  
CG\_cloud: null  
SC\_apptemp: 23.65  
SC\_cloud: 0  
being fed into the influxdb node.  
 ~~  
_Edit - not possible. Cannot post null values to influx_

> [@moebius](#):
>
> the db could be normalized and you just send the weather data without joining

Until tonight, I did send the data without joining, but it makes it more difficult to analyse in Grafana, as the data points would not be aligned when hovering over.

![align](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/2X/6/6fb730a9e6a1fb751b0c58f36d3e64a06f153068.png)

---

<div class="post-metadata">

**Author:** ![moebius](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/moebius/32/2872_2.png) [@moebius](https://discourse.nodered.org/u/moebius)\
**Post date:** [25 June 2019 20:48 UTC](https://discourse.nodered.org/t/joining-weather-data-from-2-locations/12589/4 "2019-06-25T20:48:08Z")

</div>

ic...sorry, dont know how to do that

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [26 June 2019 06:53 UTC](https://discourse.nodered.org/t/joining-weather-data-from-2-locations/12589/5 "2019-06-26T06:53:12Z")

</div>

The join node won’t know how to get initialised with nulls so I would think building your own join with a function node would be easiest.

---

<div class="post-metadata">

**Author:** ![Paul-Reed](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/paul-reed/32/66906_2.png) [@Paul-Reed](https://discourse.nodered.org/u/Paul-Reed)\
**Post date:** [26 June 2019 08:17 UTC](https://discourse.nodered.org/t/joining-weather-data-from-2-locations/12589/6 "2019-06-26T08:17:32Z")

</div>

> [@dceejay](#):
>
> building your own join with a function node would be easiest.

Hmm, that's going to be fun 🙄

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [26 June 2019 08:21 UTC](https://discourse.nodered.org/t/joining-weather-data-from-2-locations/12589/7 "2019-06-26T08:21:10Z")

</div>

Ok - then - maybe the formatter can insert the null data if needs be ?

```
CG_apptemp: cala.CG_apptemp || null;

```

etc

(of course that simple version will break if temp is exactly 0 😉

---

<div class="post-metadata">

**Author:** ![Paul-Reed](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/paul-reed/32/66906_2.png) [@Paul-Reed](https://discourse.nodered.org/u/Paul-Reed)\
**Post date:** [26 June 2019 08:53 UTC](https://discourse.nodered.org/t/joining-weather-data-from-2-locations/12589/8 "2019-06-26T08:53:22Z")

</div>

> [@dceejay](#):
>
> (of course that simple version will break if temp is exactly 0 😉

Yep, that's what I found when I tried that earlier. The cloud cover was 0, which was interpreted as a null.

> [@Paul-Reed](#):
>
> I've tried a few things like reworking the array as;
> 
> ```auto
> CG_apptemp: msg.payload.calagaldana.CG_apptemp || null, 
> 
> ```
> 
> Which really didn't work out!

I'm also wondering if the `influxdb` node can handle a null value anyway, as when it tried to send the data containing a null, I got a 400 bad request error from the influxdb node.

That being the case, maybe instead of sending;

```auto
CG_apptemp: 30.17
CG_cloud: 0
SC_apptemp: null
SC_cloud: null

```

I should be just sending;

```auto
CG_apptemp: 30.17
CG_cloud: 0

```

In fact - just read a git issue response by @mike in which he confirms this;

> ...as I understand it, influxdb 1.0 and later does not support sending a point with null values. If you want to indicate that a value is missing, one suggested technique is that you use another field to indicate missing data.

---

<div class="post-metadata">

**Author:** ![shrickus](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/shrickus/32/517_2.png) [@shrickus](https://discourse.nodered.org/u/shrickus)\
**Post date:** [26 June 2019 12:41 UTC](https://discourse.nodered.org/t/joining-weather-data-from-2-locations/12589/9 "2019-06-26T12:41:12Z")

</div>

Paul,

I would use a `change` node after your join node -- using a JSONata expression to build the influxdb input data structure by merging **whatever** objects it receives:

```auto
[
	$merge(payload),
	{
    	"metrics": "weather"
	}
]

```

This will only work if each object in your payload array has unique key names (`CG_apptemp` vs. `SC_apptemp`) -- so that looks to be fine for this use case.

The nice thing about JSONata expressions is that the output silently drops any null objects. If there is a chance that some of the data **values** will be null, we can also process them out, but the `$merge()` function will not do that for you automatically I don't believe.

---

<div class="post-metadata">

**Author:** ![Paul-Reed](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/paul-reed/32/66906_2.png) [@Paul-Reed](https://discourse.nodered.org/u/Paul-Reed)\
**Post date:** [26 June 2019 18:58 UTC](https://discourse.nodered.org/t/joining-weather-data-from-2-locations/12589/10 "2019-06-26T18:58:04Z")

</div>

> [@shrickus](#):
>
> The nice thing about JSONata expressions is that the output silently drops any null objects.

It doesn't seem to drop the null values, see the brief flow below;

```auto
[{"id":"532e211d.7cdb9","type":"function","z":"c53060.842a0fa","name":"test data","func":"msg.payload = {\n CG_apptemp: 28,\n CG_cloud: 0,\n SC_apptemp: null,\n SC_cloud: null\n};\n\nreturn msg;\n","outputs":1,"noerr":0,"x":410,"y":1120,"wires":[["4a8794d2.3c8aec"]]},{"id":"7eb46d1.98d7494","type":"debug","z":"c53060.842a0fa","name":"","active":true,"console":"false","complete":"false","x":740,"y":1120,"wires":[]},{"id":"4a8794d2.3c8aec","type":"change","z":"c53060.842a0fa","name":"merge data","rules":[{"t":"set","p":"payload","pt":"msg","to":"[\t\t$merge(payload),\t\t{\t \t\"metrics\": \"weather\"\t\t}\t]","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":570,"y":1120,"wires":[["7eb46d1.98d7494"]]},{"id":"13605a8b.89aab5","type":"inject","z":"c53060.842a0fa","name":"","topic":"","payload":"","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":"","x":260,"y":1120,"wires":[["532e211d.7cdb9"]]}]

```

Also, as the join node creates a key/value object;

![key](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/2X/8/8072fe1b75b2adf063cdb8868305dfaac316f3a0.jpeg)

it does not get processed by the JSONata merge.

The complete flow with the JSONata merge node added is;

```auto
[{"id":"c5ba0375.379","type":"darksky","z":"c53060.842a0fa","darksky":"","name":"Cala Galdana","lon":"3.960460","lat":"39.938713","date":"","time":"","mode":"node","lang":"en","units":"uk2","x":308,"y":840,"wires":[["d1c36626.4042e8"]]},{"id":"73043ed0.c3f35","type":"inject","z":"c53060.842a0fa","name":"2pm every day","topic":"","payload":"","payloadType":"str","repeat":"","crontab":"0 0-23 * * *","once":false,"onceDelay":"","x":120,"y":840,"wires":[["c5ba0375.379","80b98bb1.b56a28"]]},{"id":"e2505289.f076a","type":"darksky","z":"c53060.842a0fa","darksky":"","name":"Sa Coma","lon":"3.376063","lat":"39.579944","date":"","time":"","mode":"node","lang":"en","units":"uk2","x":330,"y":890,"wires":[["13cf9dfe.00fb62"]]},{"id":"a69c3819.a82f28","type":"join","z":"c53060.842a0fa","name":"","mode":"custom","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"60","count":"2","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":588,"y":840,"wires":[["a24646b6.b7a5c8","4b53014d.4a7c6"]]},{"id":"d1c36626.4042e8","type":"function","z":"c53060.842a0fa","name":"Parser","func":"msg.topic = \"calagaldana\";\nconst i = msg.data.currently;\n const at = i.apparentTemperature;\n const cl = i.cloudCover*100;\n\nmsg.payload = {\n CG_apptemp: at,\n CG_cloud: cl\n};\nreturn msg;\n","outputs":1,"noerr":0,"x":458,"y":840,"wires":[["a69c3819.a82f28"]]},{"id":"13cf9dfe.00fb62","type":"function","z":"c53060.842a0fa","name":"Parser","func":"msg.topic =\"sacoma\";\nconst i = msg.data.currently;\n const at = i.apparentTemperature;\n const cl = i.cloudCover*100;\n\nmsg.payload = {\n SC_apptemp: at,\n SC_cloud: cl\n};\nreturn msg;\n","outputs":1,"noerr":0,"x":460,"y":890,"wires":[["a69c3819.a82f28"]]},{"id":"80b98bb1.b56a28","type":"delay","z":"c53060.842a0fa","name":"","pauseType":"delay","timeout":"15","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"x":182,"y":890,"wires":[["e2505289.f076a"]]},{"id":"66dfa018.9ce1","type":"debug","z":"c53060.842a0fa","name":"","active":true,"console":"false","complete":"false","x":900,"y":840,"wires":[]},{"id":"a24646b6.b7a5c8","type":"change","z":"c53060.842a0fa","name":"merge data","rules":[{"t":"set","p":"payload","pt":"msg","to":"[\t\t$merge(payload),\t\t{\t \t\"metrics\": \"weather\"\t\t}\t]","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":740,"y":840,"wires":[["66dfa018.9ce1"]]},{"id":"4b53014d.4a7c6","type":"debug","z":"c53060.842a0fa","name":"","active":true,"console":"false","complete":"false","x":740,"y":890,"wires":[]}]

```

---

<div class="post-metadata">

**Author:** ![shrickus](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/shrickus/32/517_2.png) [@shrickus](https://discourse.nodered.org/u/shrickus)\
**Post date:** [26 June 2019 19:06 UTC](https://discourse.nodered.org/t/joining-weather-data-from-2-locations/12589/11 "2019-06-26T19:06:58Z")

</div>

> [@Paul-Reed](#):
>
> > The nice thing about JSONata expressions is that the output silently drops any null objects.
> 
> It doesn't seem to drop the null values, see the brief flow below;

Right, that why I said it drops **null objects** , not **null values**... and your incoming payload does not have any null objects ;\*)

Anyway, I did miss that fact that the incoming payload was a single payload object -- my bad!  
So to merge the _objects stored in the payload properties_ you need a slightly different syntax, like so:

```auto
[
	$merge(payload.*),
	{
    	"metrics": "weather"
	}
]

```

---

<div class="post-metadata">

**Author:** ![Paul-Reed](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/paul-reed/32/66906_2.png) [@Paul-Reed](https://discourse.nodered.org/u/Paul-Reed)\
**Post date:** [26 June 2019 19:25 UTC](https://discourse.nodered.org/t/joining-weather-data-from-2-locations/12589/12 "2019-06-26T19:25:46Z")

</div>

Steve, that's a great solution, and works well 😀  
If either of the weather api's do not respond, the database is now only updated with the working api result, which is exactly what I wanted.  
Thanks for the help.

Paul

---

<div class="post-metadata">

**Author:** ![Paul-Reed](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/paul-reed/32/66906_2.png) [@Paul-Reed](https://discourse.nodered.org/u/Paul-Reed)\
**Post date:** [26 June 2019 19:27 UTC](https://discourse.nodered.org/t/joining-weather-data-from-2-locations/12589/13 "2019-06-26T19:27:50Z")

</div>

If it helps anyone else in the future, here is the flow;

```auto
[{"id":"c5ba0375.379","type":"darksky","z":"c53060.842a0fa","darksky":"","name":"Cala Galdana","lon":"3.960460","lat":"39.938713","date":"","time":"","mode":"node","lang":"en","units":"uk2","x":308,"y":840,"wires":[["d1c36626.4042e8"]]},{"id":"73043ed0.c3f35","type":"inject","z":"c53060.842a0fa","name":"2pm every day","topic":"","payload":"","payloadType":"str","repeat":"","crontab":"0 0-23 * * *","once":false,"onceDelay":"","x":120,"y":840,"wires":[["c5ba0375.379","80b98bb1.b56a28"]]},{"id":"e2505289.f076a","type":"darksky","z":"c53060.842a0fa","darksky":"","name":"Sa Coma","lon":"3.376063","lat":"39.579944","date":"","time":"","mode":"node","lang":"en","units":"uk2","x":330,"y":890,"wires":[["13cf9dfe.00fb62"]]},{"id":"a69c3819.a82f28","type":"join","z":"c53060.842a0fa","name":"","mode":"custom","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"30","count":"2","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":588,"y":840,"wires":[["a24646b6.b7a5c8"]]},{"id":"d1c36626.4042e8","type":"function","z":"c53060.842a0fa","name":"Parser","func":"msg.topic = \"calagaldana\";\nconst i = msg.data.currently;\n const at = i.apparentTemperature;\n const cl = i.cloudCover*100;\n\nmsg.payload = {\n CG_apptemp: at,\n CG_cloud: cl\n};\nreturn msg;\n","outputs":1,"noerr":0,"x":458,"y":840,"wires":[["a69c3819.a82f28"]]},{"id":"13cf9dfe.00fb62","type":"function","z":"c53060.842a0fa","name":"Parser","func":"msg.topic =\"sacoma\";\nconst i = msg.data.currently;\n const at = i.apparentTemperature;\n const cl = i.cloudCover*100;\n\nmsg.payload = {\n SC_apptemp: at,\n SC_cloud: cl\n};\nreturn msg;\n","outputs":1,"noerr":0,"x":460,"y":890,"wires":[["a69c3819.a82f28"]]},{"id":"80b98bb1.b56a28","type":"delay","z":"c53060.842a0fa","name":"","pauseType":"delay","timeout":"10","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"x":182,"y":890,"wires":[["e2505289.f076a"]]},{"id":"a24646b6.b7a5c8","type":"change","z":"c53060.842a0fa","name":"merge data","rules":[{"t":"set","p":"payload","pt":"msg","to":"[\t\t$merge(payload.*),\t\t{\t \t\"metrics\": \"weather\"\t\t}\t]","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":740,"y":840,"wires":[[]]}]

```