Working FTP node ? or direct XML input

General
1

Guys,

Wanting to grab some FTP files - see here ftp://ftp2.bom.gov.au/anon/gen/fwo/IDN10064.xml

And put them into node red, so i can try and parse some Sydney weather info from our BOM.

I have looked at each of the FTP nodes and also the forum discussions and it does not seem there is a consensus on a working FTP anonymous node - does anyone have a reliable one they would recommend.

Alternatively a way to pull that XML stream directly into NR and push to a JSON stream so i can drill down and extract what i want ?

regards

Craig

2

Someone already did this in the past. There is a working solution in the link::

https://groups.google.com/d/topic/node-red/JLiYwcEiJj0/discussion

3

Thanks Andrei - my google Fu does not seem to be strong at the moment !

Craig

1 Like
4

Maybe not so fast - so going through that - they end up giving up on the FTP node as it is broken and instead head down the curl path.

The last message posted though has a flow in it - talking about jsonata - when i paste that and run it i get an error

image

And here is the flow that produces it.

[{"id":"270fb26c.d03b6e","type":"inject","z":"4c1ba2a9.c1050c","name":"Additional Info","topic":"","payload":"ftp://ftp.bom.gov.au/anon/gen/fwo/IDV10753.xml","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":"","x":410,"y":600,"wires":[["fe0a2149.3aa49"]]},{"id":"fe0a2149.3aa49","type":"exec","z":"4c1ba2a9.c1050c","command":"curl","addpay":true,"append":"","useSpawn":"false","timer":"","name":"","x":540,"y":600,"wires":[["54df56f4.4b8218"],[],[]]},{"id":"54df56f4.4b8218","type":"xml","z":"4c1ba2a9.c1050c","name":"","attr":"","chr":"","x":670,"y":580,"wires":[["8fe62ffa.54b33"]]},{"id":"de2110e4.c03fa","type":"debug","z":"4c1ba2a9.c1050c","name":"","active":true,"console":"false","complete":"payload","x":1010,"y":580,"wires":[]},{"id":"8fe62ffa.54b33","type":"change","z":"4c1ba2a9.c1050c","name":"Get Forcast","rules":[{"t":"move","p":"payload.product.forecast[0].area[52].forecast-period","pt":"msg","to":"weather","tot":"msg"},{"t":"set","p":"payload","pt":"msg","to":"{   \t\"today\":{\t        \"date\":msg.weather[0].\"$\".\"start-time-local\" ,\t        msg.weather[0].text[1].\"$\".type:product.forecast[0].area[52].\"forecast-period\"[0].text[1].\"_\"\t    },\t\"tomorrow\": {\t        \"date\": msg.weather[1].\"$\".\"start-time-local\",\t        msg.weather[1].element[1].\"$\".type: msg.weather[1].element[1].\"_\",\t        msg.weather[1].element[2].\"$\".type: msg.weather[1].element[2].\"_\" & \" degrees\",\t        msg.weather[1].element[3].\"$\".type: msg.weather[1].element[3].\"_\" & \" degrees\",\t        msg.weather[1].text[1].\"$\".type: msg.weather[1].text[1].\"_\" & \" chance of rain\"\t    },\t\"two\": {\t        \"date\": msg.weather[2].\"$\".\"start-time-local\",\t        msg.weather[2].element[1].\"$\".type: msg.weather[2].element[1].\"_\",\t        msg.weather[2].element[2].\"$\".type: msg.weather[2].element[2].\"_\" & \" degrees\",\t        msg.weather[2].element[3].\"$\".type: msg.weather[2].element[3].\"_\" & \" degrees\",\t        msg.weather[2].text[1].\"$\".type: msg.weather[2].text[1].\"_\" & \" chance of rain\"\t        },\t\"three\": {\t        \"date\": msg.weather[3].\"$\".\"start-time-local\",\t        msg.weather[3].element[1].\"$\".type: msg.weather[3].element[1].\"_\",\t        msg.weather[3].element[2].\"$\".type: msg.weather[3].element[2].\"_\" & \" degrees\",\t        msg.weather[3].element[3].\"$\".type: msg.weather[3].element[3].\"_\" & \" degrees\",\t        msg.weather[3].text[1].\"$\".type: msg.weather[3].text[1].\"_\" & \" chance of rain\"\t        },\t\"four\": {\t        \"date\": msg.weather[4].\"$\".\"start-time-local\",\t        msg.weather[4].element[1].\"$\".type: msg.weather[4].element[1].\"_\",\t        msg.weather[4].element[2].\"$\".type: msg.weather[4].element[2].\"_\" & \" degrees\",\t        msg.weather[4].element[3].\"$\".type: msg.weather[4].element[3].\"_\" & \" degrees\",\t        msg.weather[4].text[1].\"$\".type: msg.weather[4].text[1].\"_\" & \" chance of rain\"\t},\t\"five\": {\t        \"date\": msg.weather[5].\"$\".\"start-time-local\",\t        msg.weather[5].element[1].\"$\".type: msg.weather[5].element[1].\"_\",\t        msg.weather[5].element[2].\"$\".type: msg.weather[5].element[2].\"_\" & \" degrees\",\t        msg.weather[5].element[3].\"$\".type: msg.weather[5].element[3].\"_\" & \" degrees\",\t        msg.weather[5].text[1].\"$\".type: msg.weather[5].text[1].\"_\" & \" chance of rain\"\t        },\t\"six\":  {\t        \"date\": msg.weather[6].\"$\".\"start-time-local\",\t        msg.weather[6].element[1].\"$\".type: msg.weather[6].element[1].\"_\",\t        msg.weather[6].element[2].\"$\".type: msg.weather[6].element[2].\"_\" & \" degrees\",\t        msg.weather[6].element[3].\"$\".type: msg.weather[6].element[3].\"_\" & \" degrees\",\t        msg.weather[6].text[1].\"$\".type: msg.weather[6].text[1].\"_\" & \" chance of rain\"\t}\t}","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":830,"y":580,"wires":[["de2110e4.c03fa"]]}]

Guess i will rip it apart and see if i can see anything

I know nothing abut jsonata - is there an easy expression builder etc ??

Craig

5

This is exactly what you should do. I see that the JSONata expression is trying to read some elements that does not exist in the input JSON data. I removed a few ones from the change rule and it worked. You will need to compare that data received from the API against with the JSONata expression to see what fix is necessary.

r-01

[{"id":"2e7939ce.b5aec6","type":"tab","label":"Flow 2","disabled":false,"info":""},{"id":"d57fbd39.268b4","type":"inject","z":"2e7939ce.b5aec6","name":"Additional Info","topic":"","payload":"ftp://ftp.bom.gov.au/anon/gen/fwo/IDV10753.xml","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":"","x":170,"y":160,"wires":[["68616787.fc9de8"]]},{"id":"68616787.fc9de8","type":"exec","z":"2e7939ce.b5aec6","command":"curl","addpay":true,"append":"","useSpawn":"false","timer":"","name":"","x":300,"y":160,"wires":[["809d8a68.26d8e8"],[],[]]},{"id":"809d8a68.26d8e8","type":"xml","z":"2e7939ce.b5aec6","name":"","attr":"","chr":"","x":430,"y":140,"wires":[["2a45783b.a03f68","518ff716.915678"]]},{"id":"a7fe50ee.4daa8","type":"debug","z":"2e7939ce.b5aec6","name":"","active":true,"console":"false","complete":"payload","x":770,"y":140,"wires":[]},{"id":"2a45783b.a03f68","type":"change","z":"2e7939ce.b5aec6","name":"Get Forcast","rules":[{"t":"move","p":"payload.product.forecast[0].area[52].forecast-period","pt":"msg","to":"weather","tot":"msg"},{"t":"set","p":"payload","pt":"msg","to":"{\t   \"today\":{\t       \"date\":msg.weather[0].\"$\".\"start-time-local\" ,\t       msg.weather[0].text[1].\"$\".type:product.forecast[0].area[52].\"forecast-period\"[0].text[1].\"_\"\t    \t   },\t   \"tomorrow\": {\t       \"date\": msg.weather[1].\"$\".\"start-time-local\",\t       msg.weather[1].element[1].\"$\".type: msg.weather[1].element[1].\"_\",\t       msg.weather[1].element[2].\"$\".type: msg.weather[1].element[2].\"_\" & \" degrees\",\t       msg.weather[1].element[3].\"$\".type: msg.weather[1].element[3].\"_\" & \" degrees\",\t       msg.weather[1].text[1].\"$\".type: msg.weather[1].text[1].\"_\" & \" chance of rain\"\t    \t   },\t   \"two\": {\t       \"date\": msg.weather[2].\"$\".\"start-time-local\",\t       msg.weather[2].element[1].\"$\".type: msg.weather[2].element[1].\"_\",\t       msg.weather[2].element[2].\"$\".type: msg.weather[2].element[2].\"_\" & \" degrees\",\t       msg.weather[2].element[3].\"$\".type: msg.weather[2].element[3].\"_\" & \" degrees\",\t       msg.weather[2].text[1].\"$\".type: msg.weather[2].text[1].\"_\" & \" chance of rain\"\t        \t   }\t   \t   }","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":590,"y":140,"wires":[["a7fe50ee.4daa8"]]},{"id":"518ff716.915678","type":"debug","z":"2e7939ce.b5aec6","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":600,"y":200,"wires":[]}]
1 Like
6

Thanks for that - i will put in some time tonight and try to work out what this is all doing ! (and more importantly how it does it !)

Craig

7

That JSONata expression is much more complicated than it needs to be, just to provide some named fields ("today", "tomorrow", "two", etc). The only difference between each day's object (other than today) is the index number into the array.

Instead, I suggest mapping the 7-element forecast-period array, using this expression:

(
    $weather := payload.product.forecast[0].area[52]."forecast-period";
    $date := $weather.{
       "date": $."$"."start-time-local"
    };
    $elem := $weather.[element].{
       $."$".type: $."_" & ($."$".units ? "°" & $."$".units.$substring(0,1) : "")
    };
    $text := $weather.[text].{
       $."$".type: $."_"
    };
    $zip($date, $elem, $text).$merge($)
)

This produces a 7 day array of forecast objects -- which looks like this:

[
  {
    "date": "2018-09-18T05:00:00+10:00",
    "forecast_icon_code": "11",
    "precipitation_range": "1 to 2 mm",
    "air_temperature_maximum": "20°C",
    "precis": "Shower or two. Windy.",
    "probability_of_precipitation": "70%"
  },
  {
    "date": "2018-09-19T00:00:00+10:00",
    "forecast_icon_code": "11",
    "precipitation_range": "1 to 2 mm",
    "air_temperature_minimum": "7°C",
    "air_temperature_maximum": "15°C",
    "precis": "Showers.",
    "probability_of_precipitation": "80%"
  },
  {
    "date": "2018-09-20T00:00:00+10:00",
    "forecast_icon_code": "3",
    "precipitation_range": "0 to 0.2 mm",
    "air_temperature_minimum": "6°C",
    "air_temperature_maximum": "16°C",
    "precis": "Partly cloudy.",
    "probability_of_precipitation": "30%"
  },
  {
    "date": "2018-09-21T00:00:00+10:00",
    "forecast_icon_code": "3",
    "air_temperature_minimum": "8°C",
    "air_temperature_maximum": "18°C",
    "precis": "Partly cloudy.",
    "probability_of_precipitation": "5%"
  },
  {
    "date": "2018-09-22T00:00:00+10:00",
    "forecast_icon_code": "17",
    "precipitation_range": "0 to 1 mm",
    "air_temperature_minimum": "9°C",
    "air_temperature_maximum": "20°C",
    "precis": "Possible late shower.",
    "probability_of_precipitation": "40%"
  },
  {
    "date": "2018-09-23T00:00:00+10:00",
    "forecast_icon_code": "11",
    "precipitation_range": "0 to 1 mm",
    "air_temperature_minimum": "8°C",
    "air_temperature_maximum": "14°C",
    "precis": "Shower or two.",
    "probability_of_precipitation": "50%"
  },
  {
    "date": "2018-09-24T00:00:00+10:00",
    "forecast_icon_code": "3",
    "air_temperature_minimum": "6°C",
    "air_temperature_maximum": "16°C",
    "precis": "Partly cloudy.",
    "probability_of_precipitation": "5%"
  }
]

So the msg.payload[0] object represents today, msg.payload[1] is one day from now or tomorrow, and so on. The UI can add the human-readable day offset labels, or extra wording.

8

Thank you very much - that will give me more to compare with - is there a simple discussion/document on using jsonata like this - obviously i can understand the result of what you are doing - but the mechanics of geting there is another matter.

I assume that we are essentially specifing variable objects such as $weather and then trying to populate those based on looking inside the XML - which it appears we are treating as Arrays (such as area[52].

But then looking at something like the $elem is just blurring before my eyes !!

Looks like lots of reading to do here

Craig

9

The cookbook should have some ideas. I know there were some suggestions in the cookbook WIKI, looks like Steve's example would be a great one to add to that (with some annotation maybe :slight_smile:)

I suspect that, outside of IBM, Steve may be the best expert on JSONata!

Actually, I nearly forgot, I put all of my examples that were in the WIKI into a knowledgebase post on my blog:

I might nick this one too!

10

Thanks for that update. I went to the jsonata.org website and played around with their stuff and have a bit of an idea now.

Stilll struggling to understand how the stuff comes from the xml and then gets transformed - will continue reading and playing with the data that i have until i crack it

Craig

11

OK obviously i am missing something basic here. I assume even as good as Steve is - he can not look at that XLM output and know where the various arrays are going to be etc.

There must be a way that someone takes that XML string and then plays with it interactively to identify the areas and values of interest ?

Is that just through the debug node and then grabbing the path in the debug window or is there some other fancy tool that helps with this and hence then makes it easier to build the jsonata ?

Craig

12

If you play withe change node - jsonata option - you can hit the three dots on the right to open the jsonata editor and tester

image
image.png932×224 5.2 KB

This allows you to paste in your expression and then some test data and see the result as you play.

13

As Dave mentioned, I rely on the JSONata tester found in the change node pretty heavily. First, use the debug sidebar buttons to copy the value of the payload to the clipboard, then paste it into the tester...

However, your data was so big that it was easier to use the online tester at try.jsonata.org

image
image.png1205×634 52.3 KB

And you are right -- there's no way to just look at your data and know how to access the fields. For that, I rely on the debug sidebar again -- when you hover the mouse on the data you want, the "copy path" button appears on the right. Your data structure is made extra complicated because it was converted to a JS object from XML, so I would NOT suggest trying to learn JSONata with this data!

14

Aaah - i thought there was a magic step i was missing !!!

Thanks for the explanation - i will try it with the real data for the actual area i am interested in and see if i can get any further

regards

Craig

15

Would it be easier to leave it as XML or will the Jsonata tool not accept that ?

Craig

16

Uh, that would be the XMLonata tool ;*) ... if there was one

But if you really want to process XML instead, skip the xml and change nodes, install node-red-contrib-xsltransform and write some xslt rules...

17

OH right - so no is the answer !!! Already opened up one can of worms - dont need a second one.

I will knuckle down over the next couple of days and see what i can nut out

thanks for all your help Steve

Craig

18

Maybe one day we will get a XSLT node (the XML equivalent of JSONata). But most of us are using JSON over XML these days. We also have the XML node that makes a valiant attempt at turning XML into JSON.

19

I hope not Julian - it is enough with jsonata !!

The XML node appears to be swallowing this stuff and spitting out something that jsonata can work with so i am slowly pushing my way through !

Craig