Reduce large JSON and publish over http endpoint

Im having large JSON (darksky api - https://darksky.net/dev/docs) which is too large to process with esp8266 microcontroller. All i need from that JSON is timestamp and temperature from hourly > data.

I've tried many ways, but always failed.

Do you have a small sample of the data you have, and an example of what you want to convert it to?

1 Like
{
"latitude": 48.778194,
"longitude": 18.618188,
"timezone": "Europe/Bratislava",
"currently": {
"time": 1575386648,
"summary": "Clear",
"icon": "clear-night",
"precipIntensity": 0,
"precipProbability": 0,
"temperature": 0.36,
"apparentTemperature": -4.29,
"dewPoint": -3.74,
"humidity": 0.74,
"pressure": 1027.9,
"windSpeed": 4.68,
"windGust": 4.68,
"windBearing": 357,
"cloudCover": 0.01,
"uvIndex": 0,
"visibility": 10.52,
"ozone": 275.9
},
"hourly": {
"summary": "Clear throughout the day.",
"icon": "clear-day",
"data": [
{
"time": 1575385200,
"summary": "Clear",
"icon": "clear-night",
"precipIntensity": 0,
"precipProbability": 0,
"temperature": 0.83,
"apparentTemperature": -3.74,
"dewPoint": -3.38,
"humidity": 0.73,
"pressure": 1027.7,
"windSpeed": 4.76,
"windGust": 4.76,
"windBearing": 356,
"cloudCover": 0.03,
"uvIndex": 0,
"visibility": 10.356,
"ozone": 275.8
},

. . . this goes on like 48 times (for every hour in next two days)
and just expecting to get

[ 
   { 
      "time":1575385200,
      "temperature":0.83
   },
   { 
      "time":1575388800,
      "temperature":-0.39
   }
]

. . . all 48 instances

You can use a Change node with a JSONata expression to do this.

Configure the node to set msg.payload to the Expression $.payload.hourly.data.{"time":time,"temperature":temperature}

2 Likes

OMG, that's great! Thank You! Now I just need to wrap arround my head about what you're writed there.

Is the propper way to publish it like this please?

[{"id":"df5b9cf2.99f9e","type":"tab","label":"Flow 1","disabled":false,"info":""},{"id":"d97a19a8.2ae4b8","type":"http request","z":"df5b9cf2.99f9e","name":"","method":"GET","ret":"obj","paytoqs":false,"url":"...../temperature/darksky.json","tls":"","persist":false,"proxy":"","authType":"","x":290,"y":180,"wires":[["208da9c4.5ce226","819aa565.7cafc8"]]},{"id":"9ee799af.2313f8","type":"inject","z":"df5b9cf2.99f9e","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":120,"y":180,"wires":[["d97a19a8.2ae4b8"]]},{"id":"819aa565.7cafc8","type":"debug","z":"df5b9cf2.99f9e","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":790,"y":180,"wires":[]},{"id":"8f6d090d.446f98","type":"http in","z":"df5b9cf2.99f9e","name":"","url":"/darksky","method":"get","upload":false,"swaggerDoc":"","x":110,"y":120,"wires":[["d97a19a8.2ae4b8"]]},{"id":"4292168a.29a608","type":"http response","z":"df5b9cf2.99f9e","name":"","statusCode":"","headers":{},"x":710,"y":120,"wires":[]},{"id":"208da9c4.5ce226","type":"change","z":"df5b9cf2.99f9e","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"$.payload.hourly.data.{\"time\":time,\"temperature\":temperature}","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":480,"y":220,"wires":[["819aa565.7cafc8","4292168a.29a608"]]}]

It complains about "No response object"

How are you triggering it when you get that error? If you are using the Inject node, then the HTTP Response node will log that error because the message hasn't originated from an HTTP In node for it to reply to.

1 Like

I saw your flow when I imported it. This is why I asked when are you seeing the No Response Object error message?

1 Like

I see, sorry. In debug window. But I've filtered the IP address... So it won't work for you. I trigger manually. Injecting timestamp.

image
Anyway, I should not really worry, because it works :smile:

So as I said in my reply, you are getting that error because a message is arriving at the HTTP Response node that did not start at the HTTP In node. That is to be expected if you are manually triggering the flow - there is no HTTP request for the HTTP Response node to reply to.

1 Like

I understand now. Thank you very much for your help.

If I may have one more question please, how do I include some data from original JSON but from currently > temperature? It always turns to disaster if I try myself...

As with your original question, its much easier if you provide an actual example of what you'd want that to look like.

1 Like

Original JSON as it was previousl and new as follows:

{ 
   "currently":{ 
      "temperature":0.36
   },
   "hourly":[ 
      { 
         "time":1575385200,
         "temperature":0.83
      },
      { 
         "time":1575388800,
         "temperature":-0.39
      }
   ]
} ......

So basically I'd like to include current temperature at the beginning of my new JSON.

Change the expression to:

{"currently":$$.payload.currently.{"temperature":temperature}, "hourly":[$$.payload.hourly.data.{"time":time,"temperature":temperature}]}
1 Like

I found out, that even more reduce this JSON will be to do it as an JSON array:

{ 
   "currently":{ 
      "temperature":1.69
   },
   "hourly":[ 
      { 
         "temperature":[ 
            1.67,
            2.98,
            3.45,
            4.42   ...
         ],
         "precipIntensity":[ 
            0.2968,
            0.4584,
            0.6258,
            0.2321   ...
         ]
      }
   ]
}

Will it be possible using this method please?

I think your JSON can be simplified a bit more as you still have hourly as an array that contains just one element - it would be better to have hourly as the object with the temperature/precipIntesity properties under it....

{
  "currently": {
    "temperature": 0.36
  },
  "hourly": {
    "temperature": [
      0.83,
      0.93,
      1.03
    ],
    "precipIntensity": [
      1,
      2,
      3
    ]
  }
}

The following expression generates that:

{"currently":$$.payload.currently.{"temperature":temperature},"hourly":{"temperature": $$.payload.hourly.data.temperature,"precipIntensity":$$.payload.hourly.data.precipIntensity}}

If you really want the output as you propose, with hourly being a single-element array, then you can use:

{"currently":$$.payload.currently.{"temperature":temperature},"hourly":[{"temperature": $$.payload.hourly.data.temperature,"precipIntensity":$$.payload.hourly.data.precipIntensity}]}
1 Like