Parse gpx file and extract fields from javascript object

#1

Hi,

I am trying to parse a gpx file and filter out only specific sets of information. I have read lots of different approaches on the forum but I'm pretty sure there are much easier ways than my current "dogs dinner" set of flows.

To start here is a simplified extract of my gpx file (in reality I have many of these files and they contain many more entries than you see here)

I would like to end up with a message payload which has the following elements from this file:

time
name
link
type
lat
lon
ele

I would like this data ordered in time order. This is all the pre-processing steps I need to do before I put it into a influx db.

Where I have got to so far is:

  1. a file in node which successfuly brings in the file which is stored on my local drive - think I am good here
  2. An edit xml node which takes the file contents and makes it into a javascript object - I suspect this is also ok
  3. Once I get this far I need to extract these fields out of the javascript object which is where I fall down. I have tried to use some change nodes and split nodes to access the arrays but without success. Here is a screenshot of the debug node after I get the object out of the xml node in step 2 above:

Any help would be appreciated.

0 Likes

#2

You advanced quite well on your endeavor. It would be helpful to know how the final data structure should look like, for each case. For instance, for the waypoints: the resulting data structure should be an array or an object? Need details in order to know how to extract and format the data.

Also helps to share a dataset so people in the forum can play and test. The site try.jsonata.org allows you to save a dataset and share the link. I saved a generic .gpx file for testing.

http://try.jsonata.org/rycjYmSX4

http://try.jsonata.org/SkOZ0XHX4

0 Likes

#3

Hi Andrei

Thanks. I did a bit of research on jsonata but when I try to put my file into the site it doesnt fit the requirements as it is still in gpx format and not in json format. Attached is the sample gpx file if that helps.

2018-08-06.txt (1.4 KB)

As far as the resulting data structure is concerned I need to see a very simple table structure as follows:

The structure is in columns A through G. On the right of that table I have given some further comments where you can see that the first line represents a waypoint from the file. while the next three are tracks. The first track has only 1 trackpoint whole the other has 2 trackpoints.

Regards
Gavin

0 Likes

#4

HI Gavin,

Could move a little bit, but did not test thoroughly (one single test performed). Did not revise the code/flow either. Looks promising though.

[{"id":"6b3a6c54.8944a4","type":"tab","label":"Flow 1","disabled":false,"info":""},{"id":"b3646d87.5c365","type":"inject","z":"6b3a6c54.8944a4","name":"Go","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":110,"y":100,"wires":[["7d72674a.daa2c8"]]},{"id":"7d72674a.daa2c8","type":"change","z":"6b3a6c54.8944a4","name":"Dataset","rules":[{"t":"set","p":"payload","pt":"msg","to":"{\"gpx\":{\"$\":{\"xmlns:xsi\":\"http://www.w3.org/2001/XMLSchema-instance\",\"version\":\"1.1\",\"xmlns\":\"http://www.topografix.com/GPX/1/1\",\"creator\":\"Arc App\"},\"wpt\":[{\"$\":{\"lat\":\"55.7426809446821\",\"lon\":\"12.5387628551551\"},\"time\":[\"2018-08-05T18:12:17+02:00\"],\"ele\":[\"24.1086582523434\"],\"name\":[\"Nick & Beatrice\"]},{\"$\":{\"lat\":\"55.7426703800558\",\"lon\":\"12.5386267996253\"},\"time\":[\"2018-08-06T19:16:45+02:00\"],\"ele\":[\"23.6186420128157\"],\"name\":[\"Nick & Beatrice\"]}],\"trk\":[{\"type\":[\"cycling\"],\"trkseg\":[{\"trkpt\":[{\"$\":{\"lat\":\"55.7445792136026\",\"lon\":\"12.5378666216902\"},\"ele\":[\"18.6788705537672\"],\"time\":[\"2018-08-06T06:44:31+02:00\"]}]}]},{\"type\":[\"running\"],\"trkseg\":[{\"_\":\":\\r\\t\\t\\t\\r\\t\\t\\t\\r\\t\\t\",\"trkpt\":[{\"$\":{\"lat\":\"55.7496287606377\",\"lon\":\"12.5361206138927\"},\"ele\":[\"14.785813716646\"],\"time\":[\"2018-08-06T06:53:13+02:00\"]},{\"$\":{\"lat\":\"55.7497410046034\",\"lon\":\"12.5360187042551\"},\"ele\":[\"15.529407871697\"],\"time\":[\"2018-08-06T06:53:20+02:00\"]}]}]},{\"type\":[\"walking\"],\"trkseg\":[{\"trkpt\":[{\"$\":{\"lat\":\"55.7434584900594\",\"lon\":\"12.5358218184678\"},\"ele\":[\"16.52253667917\"],\"time\":[\"2018-08-06T19:12:48+02:00\"]},{\"$\":{\"lat\":\"55.7434573815026\",\"lon\":\"12.5358484728118\"},\"ele\":[\"16.4778901219751\"],\"time\":[\"2018-08-06T19:12:57+02:00\"]}]}]}]}}","tot":"json"}],"action":"","property":"","from":"","to":"","reg":false,"x":300,"y":100,"wires":[["96befdae.d1c3a","57e7d42e.d3db0c"]]},{"id":"96befdae.d1c3a","type":"change","z":"6b3a6c54.8944a4","name":"Waypoint","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.gpx.wpt","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":480,"y":100,"wires":[["d66406a3.716808"]]},{"id":"d66406a3.716808","type":"split","z":"6b3a6c54.8944a4","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"topic","x":650,"y":100,"wires":[["37eaabdb.4d8fc4"]]},{"id":"37eaabdb.4d8fc4","type":"function","z":"6b3a6c54.8944a4","name":"Format string","func":"let pay = msg.payload;\n\nlet time = [...pay.time];\nlet name = [...pay.name];\nlet lat  = [pay[\"$\"].lat];\nlet lon  = [pay[\"$\"].lon];\nlet ele  = [...pay.ele];\n\n\nmsg.payload = `${time} , ${name} , waypoint , ${lat} ,${lon} , ${ele}`;\n\nreturn msg;","outputs":1,"noerr":0,"x":810,"y":100,"wires":[["19b2577e.3f5889"]]},{"id":"57e7d42e.d3db0c","type":"change","z":"6b3a6c54.8944a4","name":"Track","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.gpx.trk","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":150,"y":160,"wires":[["a214ff2.33f9d"]]},{"id":"a214ff2.33f9d","type":"split","z":"6b3a6c54.8944a4","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":270,"y":160,"wires":[["99ede5dc.eb3aa8"]]},{"id":"99ede5dc.eb3aa8","type":"function","z":"6b3a6c54.8944a4","name":"Format track","func":"msg.payload = { [msg.payload.type[0]] : msg.payload.trkseg[0].trkpt };\n\nreturn msg;","outputs":1,"noerr":0,"x":410,"y":160,"wires":[["f24c9edb.b2165"]]},{"id":"f24c9edb.b2165","type":"split","z":"6b3a6c54.8944a4","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"topic","x":550,"y":160,"wires":[["85acec62.df1d5"]]},{"id":"85acec62.df1d5","type":"split","z":"6b3a6c54.8944a4","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"topic","x":670,"y":160,"wires":[["230b7ea8.df3322"]]},{"id":"230b7ea8.df3322","type":"function","z":"6b3a6c54.8944a4","name":"Format string","func":"let pay = msg.payload;\n\nlet time = [...pay.time];\nlet lat  = [pay[\"$\"].lat];\nlet lon  = [pay[\"$\"].lon];\nlet ele  = [...pay.ele];\nlet type = msg.topic;\n\nmsg.payload = `${time} , no-name, ${type} , ${lat} ,${lon} , ${ele}`;\n\nreturn msg;","outputs":1,"noerr":0,"x":810,"y":160,"wires":[["19b2577e.3f5889"]]},{"id":"19b2577e.3f5889","type":"file","z":"6b3a6c54.8944a4","name":"","filename":"C:/users/OCM/.node-red/static/nrfiles/gps3.csv","appendNewline":true,"createDir":false,"overwriteFile":"false","x":1100,"y":100,"wires":[[]]}]
0 Likes

#5

I did a change in the output of the flow by adding a name to each track segment. Apparently, all works well, at least for the given example dataset. Again, I found very interesting handling a .gpx file (wanted to do since some time but never got the chance / motivation).

[{"id":"6b3a6c54.8944a4","type":"tab","label":"Flow 1","disabled":false,"info":""},{"id":"b3646d87.5c365","type":"inject","z":"6b3a6c54.8944a4","name":"Go","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":110,"y":80,"wires":[["76869e6b.c5d54"]]},{"id":"7d72674a.daa2c8","type":"change","z":"6b3a6c54.8944a4","name":"Dataset","rules":[{"t":"set","p":"payload","pt":"msg","to":"{\"gpx\":{\"$\":{\"xmlns:xsi\":\"http://www.w3.org/2001/XMLSchema-instance\",\"version\":\"1.1\",\"xmlns\":\"http://www.topografix.com/GPX/1/1\",\"creator\":\"Arc App\"},\"wpt\":[{\"$\":{\"lat\":\"55.7426809446821\",\"lon\":\"12.5387628551551\"},\"time\":[\"2018-08-05T18:12:17+02:00\"],\"ele\":[\"24.1086582523434\"],\"name\":[\"Nick & Beatrice\"]},{\"$\":{\"lat\":\"55.7426703800558\",\"lon\":\"12.5386267996253\"},\"time\":[\"2018-08-06T19:16:45+02:00\"],\"ele\":[\"23.6186420128157\"],\"name\":[\"Nick & Beatrice\"]}],\"trk\":[{\"type\":[\"cycling\"],\"trkseg\":[{\"trkpt\":[{\"$\":{\"lat\":\"55.7445792136026\",\"lon\":\"12.5378666216902\"},\"ele\":[\"18.6788705537672\"],\"time\":[\"2018-08-06T06:44:31+02:00\"]}]}]},{\"type\":[\"running\"],\"trkseg\":[{\"_\":\":\\r\\t\\t\\t\\r\\t\\t\\t\\r\\t\\t\",\"trkpt\":[{\"$\":{\"lat\":\"55.7496287606377\",\"lon\":\"12.5361206138927\"},\"ele\":[\"14.785813716646\"],\"time\":[\"2018-08-06T06:53:13+02:00\"]},{\"$\":{\"lat\":\"55.7497410046034\",\"lon\":\"12.5360187042551\"},\"ele\":[\"15.529407871697\"],\"time\":[\"2018-08-06T06:53:20+02:00\"]}]}]},{\"type\":[\"walking\"],\"trkseg\":[{\"trkpt\":[{\"$\":{\"lat\":\"55.7434584900594\",\"lon\":\"12.5358218184678\"},\"ele\":[\"16.52253667917\"],\"time\":[\"2018-08-06T19:12:48+02:00\"]},{\"$\":{\"lat\":\"55.7434573815026\",\"lon\":\"12.5358484728118\"},\"ele\":[\"16.4778901219751\"],\"time\":[\"2018-08-06T19:12:57+02:00\"]}]}]}]}}","tot":"json"}],"action":"","property":"","from":"","to":"","reg":false,"x":800,"y":80,"wires":[[]]},{"id":"96befdae.d1c3a","type":"change","z":"6b3a6c54.8944a4","name":"Waypoint","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.gpx.wpt","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":140,"y":180,"wires":[["d66406a3.716808"]]},{"id":"d66406a3.716808","type":"split","z":"6b3a6c54.8944a4","name":"Split0","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"topic","x":390,"y":180,"wires":[["37eaabdb.4d8fc4"]]},{"id":"37eaabdb.4d8fc4","type":"function","z":"6b3a6c54.8944a4","name":"Format string waypoint","func":"let pay = msg.payload;\n\nlet time = [...pay.time];\nlet name = [...pay.name];\nlet lat  = [pay[\"$\"].lat];\nlet lon  = [pay[\"$\"].lon];\nlet ele  = [...pay.ele];\n\n\nmsg.payload = `${time} , ${name} , waypoint , ${lat} ,${lon} , ${ele}`;\n\nreturn msg;","outputs":1,"noerr":0,"x":660,"y":180,"wires":[["c58e7314.2c7c1"]]},{"id":"57e7d42e.d3db0c","type":"change","z":"6b3a6c54.8944a4","name":"Track","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.gpx.trk","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":130,"y":260,"wires":[["a214ff2.33f9d"]]},{"id":"a214ff2.33f9d","type":"split","z":"6b3a6c54.8944a4","name":"Split1","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"topic","x":250,"y":260,"wires":[["99ede5dc.eb3aa8"]]},{"id":"99ede5dc.eb3aa8","type":"function","z":"6b3a6c54.8944a4","name":"Format track","func":"msg.payload = { [msg.payload.type[0]] : msg.payload.trkseg[0].trkpt };\n\nreturn msg;","outputs":1,"noerr":0,"x":390,"y":260,"wires":[["f24c9edb.b2165"]]},{"id":"f24c9edb.b2165","type":"split","z":"6b3a6c54.8944a4","name":"Split2","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"topic","x":530,"y":260,"wires":[["85acec62.df1d5"]]},{"id":"85acec62.df1d5","type":"split","z":"6b3a6c54.8944a4","name":"Split3","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"topic","x":650,"y":260,"wires":[["230b7ea8.df3322"]]},{"id":"230b7ea8.df3322","type":"function","z":"6b3a6c54.8944a4","name":"Format string track","func":"let pay = msg.payload;\n\nlet time = [...pay.time];\nlet lat  = [pay[\"$\"].lat];\nlet lon  = [pay[\"$\"].lon];\nlet ele  = [...pay.ele];\nlet ind = msg.parts.index;\nlet type = msg.topic;\n\n\nmsg.payload = `${time} , ${type}-track-${ind}, ${type} , ${lat} ,${lon} , ${ele}`;\n\nreturn msg;\n","outputs":1,"noerr":0,"x":810,"y":260,"wires":[["c58e7314.2c7c1"]]},{"id":"c58e7314.2c7c1","type":"file","z":"6b3a6c54.8944a4","name":"Save gps-1.csv","filename":"C:/users/OCM/.node-red/static/nrfiles/gps-2.csv","appendNewline":true,"createDir":false,"overwriteFile":"false","x":940,"y":180,"wires":[[]]},{"id":"76869e6b.c5d54","type":"file in","z":"6b3a6c54.8944a4","name":"Read gps-1.gpx","filename":"C:/users/OCM/.node-red/static/nrfiles/gps-2.gpx","format":"utf8","chunk":false,"sendError":false,"x":280,"y":80,"wires":[["16332d.940c1cd3"]]},{"id":"16332d.940c1cd3","type":"xml","z":"6b3a6c54.8944a4","name":"","property":"payload","attr":"","chr":"","x":470,"y":80,"wires":[["a6f40049.3baf7"]]},{"id":"a6f40049.3baf7","type":"link out","z":"6b3a6c54.8944a4","name":"","links":["c9f02adb.976ee8"],"x":575,"y":80,"wires":[]},{"id":"c9f02adb.976ee8","type":"link in","z":"6b3a6c54.8944a4","name":"","links":["a6f40049.3baf7"],"x":35,"y":220,"wires":[["96befdae.d1c3a","57e7d42e.d3db0c"]]},{"id":"81f85fbb.7666e","type":"catch","z":"6b3a6c54.8944a4","name":"","scope":null,"x":140,"y":360,"wires":[["1d66c9a7.942cf6"]]},{"id":"1d66c9a7.942cf6","type":"debug","z":"6b3a6c54.8944a4","name":"Log Error","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":290,"y":360,"wires":[]}]
0 Likes

#6

Bit of an aside. The node-red-contrib-web-worldmap should be able to load GPX files if you want to see them on a map

1 Like

#7

Hi Andrei
Thank you very much for your help on this, it looks like it works, just a couple more requests.

  1. I need to make sure I understand all of the individual steps so that I am able to learn how this stuff works. I am good on the reading of the gpx file and on converting from xml to javascript object and I am pretty sure I understand the jsonata change node and then the split into individual message. But, would you be able to give me a little bit of interpretation on the function nodes used after the splits....I am not a java expert and I presume this is standard java code within these function nodes..correct
  2. When I used an earlier version of a gpx file (rather than the one I sent you) it had a few small differences on the structure for waypoint...I tried to alter the function node to cater for this but it didnt work (although I think I came close). Perhaps you could assist

The structure in the file I sent you was:

But the new file has a slightly different structure, where there is no elevation attribute, but there is a link attribute (although the link appear slightly differently as it is not referenced in quite the same way as the other attributes)...see here:

and finally there is one more complication where sometimes there is no link and the waypoint just appears as follows:

You help would be much appreciated

Regards
Gavin

0 Likes

#8

My pleasure to help and it is great for me as I always learn a lot (after reading this post and you will understand what I mean).

I will answer point #1 below and tomorrow I will come back to point #2.

So, how the JavaScript (not Java) code works inside the functions node? Both nodes are practically the same, so I will paste below the first one for the explanation.

let pay = msg.payload;

let time = [...pay.time];
let name = [...pay.name];
let lat  = [pay["$"].lat];
let lon  = [pay["$"].lon];
let ele  = [...pay.ele];

msg.payload = `${time} , ${name} , waypoint , ${lat} ,${lon} , ${ele}`;

return msg;

The purpose here is to extract information from the object and arrays, store them into variables and then use those variables to build the desired string. This string is built with the Template Literal construction from JavaScript. The string will have five variables whose names are enclosed into $ { }. That is the easy part.

How to extract values from arrays and objects to the variables? First, it is important to note that the payload that enters the function node has this structure:

r-01

It is an object with four key/value pairs. Three of them are arrays.

The usual way to extract values from an array would be like this:

let time = msg.payload.time[0];

Sometimes this kind of expression becomes large and some people get confused with the indexes.

My personal preference is to use another capability of JavaScript that is the array destructuring, and therefore I used instead:

let time = [...pay.time];

which is a complete mistake on my side. I should have written:

let [time] = pay.time;

In the first case, the data stored in the variable is an array type, whereas in the latter case it is a string type. For this specific case, both ways will work well but they are different things (first case = array copy, second case = destructuring).

Now, missing to explain how to extract data from the object that includes latitude and longitude.

The most usual JavaScript way is :

let lat = msg.pay["$"].lat; or
let lat = msg.pay.$".lat;

I added by mistake an extra pair of square brackets. Luckily, for this specific use case, this is not a breaking mistake but should be corrected to:

let lat  = pay["$"].lat;
let lon  = pay["$"].lon;

Therefore the final code inside the function should be

let pay = msg.payload;

let [time] = pay.time;
let [name] = pay.name;
let lat  = pay["$"].lat;
let lon  = pay["$"].lon;
let [ele]  = pay.ele;

msg.payload = `${time} , ${name} , waypoint , ${lat} ,${lon} , ${ele}`;

return msg;

I hope this is clear (despite being a lengthy explanation).

2 Likes

#9

Let's go for it.

The issues show that the parser should be more robust. Before we jump to the code let me know what it is the expected outcome (final csv file) when we find a tag that was not considered before, like the <link> tag. Should it be added to a new column in the csv file or should we just disregard it?

The other situation is a non-existing tag in the file. The easiest way is to assign the value undefined to the element generating an output csv file like below. Is this acceptable or do you expect something different?

0 Likes

#10

Thanks Andrei your explanations are very clear and helpful. I guess at this stage I would go with the option of adding a new column in the output file...I would prefer to get the information and then figure out if I want it or not rather than just disregarding.

Atached is a copy of the new file if that helps.2018-06.txt (1.3 MB)

0 Likes

#11

HI Gavin,

I modified the flow to address those two issues mentioned earlier. The flow will now import any tag (waypoint and track only) that exists in the .gpx file (according to the GPX schema) and will have the data stored in a variable. It is up to you to modify the code in the function node to create the CSV file in the format you want. For that purpose, you have to modify the last assignment line in the function node.

msg.payload = `${time} , ${name} , waypoint , ${lat} , ${lon}, ${ele}`;

I did not test thoroughly and don't think I will find spare time until next week. If you find any issue that you can´t correct please let me know.

When importing the resulting CSV file to excel I see that there is something else to address. It may happen that the original file has full names stored with a separator comma. The flow is not doing any sanity check so the name will be output to the CSV file and that will add additional columns.

2018-06-01T09:05:04+02:00 , Christiansbro, Nordeas hovedkontor , waypoint , 55.67299 , 12.58789, undefined

Note that I added only a debug node for errors. I see that the testing file you provide has three lines troubled somehow. Looks like they are malformed (missing tags) but not sure.
Flow:

[{"id":"6b3a6c54.8944a4","type":"tab","label":"GPX  Parse","disabled":false,"info":"Source:https://discourse.nodered.org/t/parse-gpx-file-and-extract-fields-from-javascript-object/7151/6\n\n"},{"id":"b3646d87.5c365","type":"inject","z":"6b3a6c54.8944a4","name":"Go","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":190,"y":160,"wires":[["76869e6b.c5d54"]]},{"id":"96befdae.d1c3a","type":"change","z":"6b3a6c54.8944a4","name":"Waypoint","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.gpx.wpt","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":220,"y":240,"wires":[["d66406a3.716808"]]},{"id":"d66406a3.716808","type":"split","z":"6b3a6c54.8944a4","name":"Split0","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"topic","x":350,"y":240,"wires":[["37eaabdb.4d8fc4"]]},{"id":"37eaabdb.4d8fc4","type":"function","z":"6b3a6c54.8944a4","name":"Format  waypoint","func":"msg.pay = msg.payload;\n\nmsg.latlon = msg.payload.$;\n\n\nlet {\nele,\ntime,\nmagvar,\ngeoidheight,\nname,\ncmt,\ndesc,\nsrc,\nlink,\nsym,\ntype,\nfix,\nsat,\nhdop,\nvdop,\npdop,\nageofdgpsdata,\ndgpsid,\nextensions\n} = msg.pay;\n\nlet {lat, lon} = msg.latlon;\n\nmsg.payload = `${time} , ${name} , waypoint , ${lat} , ${lon}, ${ele}`;\n\nreturn msg;","outputs":1,"noerr":0,"x":500,"y":240,"wires":[["c58e7314.2c7c1"]]},{"id":"57e7d42e.d3db0c","type":"change","z":"6b3a6c54.8944a4","name":"Track","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.gpx.trk","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":210,"y":300,"wires":[["a214ff2.33f9d"]]},{"id":"a214ff2.33f9d","type":"split","z":"6b3a6c54.8944a4","name":"Split1","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"topic","x":350,"y":300,"wires":[["99ede5dc.eb3aa8"]]},{"id":"99ede5dc.eb3aa8","type":"function","z":"6b3a6c54.8944a4","name":"Format track","func":"msg.pay = msg.payload.trkseg[0].trkpt[0];\nmsg.latlon = msg.pay.$;\nmsg.type = msg.payload.type[0];\n\n\nlet typetrk = msg.type;\n\n\nlet {\nele,\ntime,\nmagvar,\ngeoidheight,\nname,\ncmt,\ndesc,\nsrc,\nlink,\nsym,\ntype,\nfix,\nsat,\nhdop,\nvdop,\npdop,\nageofdgpsdata,\ndgpsid,\nextensions\n} = msg.pay;\n\nlet {lat, lon} = msg.latlon;\n\nmsg.payload = `${time} , ${typetrk}-track, ${typetrk} , ${lat} ,${lon} , ${ele}`;\n\nreturn msg;","outputs":1,"noerr":0,"x":510,"y":300,"wires":[["c58e7314.2c7c1","23f11fff.8343d"]]},{"id":"c58e7314.2c7c1","type":"file","z":"6b3a6c54.8944a4","name":"Save gps-4.csv","filename":"C:/users/OCM/.node-red/static/nrfiles/gps-4.csv","appendNewline":true,"createDir":false,"overwriteFile":"false","x":740,"y":300,"wires":[[]]},{"id":"76869e6b.c5d54","type":"file in","z":"6b3a6c54.8944a4","name":"Read gps-4.gpx","filename":"C:/users/OCM/.node-red/static/nrfiles/gps-4.gpx","format":"utf8","chunk":false,"sendError":false,"x":360,"y":160,"wires":[["16332d.940c1cd3"]]},{"id":"16332d.940c1cd3","type":"xml","z":"6b3a6c54.8944a4","name":"","property":"payload","attr":"","chr":"","x":530,"y":160,"wires":[["a6f40049.3baf7"]]},{"id":"81f85fbb.7666e","type":"catch","z":"6b3a6c54.8944a4","name":"","scope":null,"x":200,"y":420,"wires":[["1d66c9a7.942cf6"]]},{"id":"1d66c9a7.942cf6","type":"debug","z":"6b3a6c54.8944a4","name":"Log Error","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":350,"y":420,"wires":[]},{"id":"b3dce694.ac8798","type":"comment","z":"6b3a6c54.8944a4","name":"Parse GPX file format","info":"","x":140,"y":80,"wires":[]},{"id":"7efbe2b0.f9919c","type":"comment","z":"6b3a6c54.8944a4","name":"waypoint","info":"<...\nlat=\"latitudeType [1] ?\"\nlon=\"longitudeType [1] ?\"> \n<ele> xsd:decimal </ele> [0..1] ?\n<time> xsd:dateTime </time> [0..1] ?\n<magvar> degreesType </magvar> [0..1] ?\n<geoidheight> xsd:decimal </geoidheight> [0..1] ?\n<name> xsd:string </name> [0..1] ?\n<cmt> xsd:string </cmt> [0..1] ?\n<desc> xsd:string </desc> [0..1] ?\n<src> xsd:string </src> [0..1] ?\n<link> linkType </link> [0..*] ?\n<sym> xsd:string </sym> [0..1] ?\n<type> xsd:string </type> [0..1] ?\n<fix> fixType </fix> [0..1] ?\n<sat> xsd:nonNegativeInteger </sat> [0..1] ?\n<hdop> xsd:decimal </hdop> [0..1] ?\n<vdop> xsd:decimal </vdop> [0..1] ?\n<pdop> xsd:decimal </pdop> [0..1] ?\n<ageofdgpsdata> xsd:decimal </ageofdgpsdata> [0..1] ?\n<dgpsid> dgpsStationType </dgpsid> [0..1] ?\n<extensions> extensionsType </extensions> [0..1] ?\n</...>","x":200,"y":520,"wires":[]},{"id":"5e437cf0.476a34","type":"comment","z":"6b3a6c54.8944a4","name":"trk","info":"<...> \n<name> xsd:string </name> [0..1] ?\n<cmt> xsd:string </cmt> [0..1] ?\n<desc> xsd:string </desc> [0..1] ?\n<src> xsd:string </src> [0..1] ?\n<link> linkType </link> [0..*] ?\n<number> xsd:nonNegativeInteger </number> [0..1] ?\n<type> xsd:string </type> [0..1] ?\n<extensions> extensionsType </extensions> [0..1] ?\n<trkseg> trksegType </trkseg> [0..*] ?\n</...>","x":350,"y":520,"wires":[]},{"id":"c9f02adb.976ee8","type":"link in","z":"6b3a6c54.8944a4","name":"","links":["a6f40049.3baf7"],"x":75,"y":240,"wires":[["96befdae.d1c3a","57e7d42e.d3db0c"]]},{"id":"a6f40049.3baf7","type":"link out","z":"6b3a6c54.8944a4","name":"","links":["c9f02adb.976ee8"],"x":635,"y":160,"wires":[]},{"id":"23f11fff.8343d","type":"debug","z":"6b3a6c54.8944a4","name":"Debug1","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":720,"y":340,"wires":[]}]

0 Likes

#12

Thanks Andrei that seems to work well, although I can see you have done it quite differently to the earlier version in the change node, specifically:

msg.pay = msg.payload;
msg.latlon = msg.payload.$;

let {
ele,
time,
magvar,
geoidheight,
name,
cmt,
desc,
src,
link,
sym,
type,
fix,
sat,
hdop,
vdop,
pdop,
ageofdgpsdata,
dgpsid,
extensions
} = msg.pay;

let {lat, lon} = msg.latlon;

msg.payload = ${time} , ${name} , waypoint , ${lat} , ${lon}, ${ele};

I presume that the different variables in the "let {...." refers to the standard variables of a gpx file...is that correct? Also the way you access these variables is different to the
temaplet literal and array destructuring you used before...is there a reason for doing it differently?

My next step is to sort the outputs from the waypoint and track nodes so that they are in dte order. I tried to use a sort node but it doesn't seem to work...any suggestions there?

Kind Regards
Gavin

0 Likes

#13

Hi Gavin, I am happy that you asked. Good questions indeed.

You are right. I changed the way to extract the data from the object and the reason is that we needed a more robust parser.

The earlier code I tried (excerpt below for this explanation) assumes that the JavaScript object stored in msg.payload :

(a) always will have those three properties :x:
(b) each of those properties has an array as its value :white_check_mark:
(c) each array has only one element. :question:

let [time] = pay.time;
let [name] = pay.name;
let [ele] = pay.ele;

Well, it turned out that assumption (a) was wrong. Later on, you have shown that sometimes the file will not have some tags (like elevation <ele> for instance). The end result is that the code will throw an error when it tries to extract data from a property that does not exist (for the code used).

Another issue you identified is that the file may contain tags that were not in your very first file, like the tag <link> for instance. The way to address this issue was to check the schema (kind of protocol description) for the GPX file. It shows all possible valid tags that may appear. So, it thought it would be better to make the code more resilient by adding the possibility to extract all possible tags. Even if you decide not to include them in your final file the data will be easily available.

We have therefore the use a new data extraction to cope with the two issues mentioned above.

I switched then to object destructuring (instead of array destructuring). Using this approach will give you all possible tags inside variable names. In case the tag does not exist no error will be thrown, instead, the variable will be assigned a value (undefined but can default to something else).

I believe we will still stumble on a couple of issues. One is the case where the original file has malformed data, which is indeed the case for the testing file you provided. This can result in issues, like being unable to sort the output by date. I believe this is why you did not succeed when trying the sort node (not your fault and not an issue with the node itself). Another potential issue is the assumption (c) explained above. If this assumption is wrong (it may be) it will be necessary to find a fix or a workaround.

0 Likes

#14

A correction word here as is not rigth use the wording "malformed" in such case (after all the XML node parsed the file correctly). It happens that your file has two empy trackings (delimited by <trkseg /> if you want to find them in the file) . Injecting an empty msg in an split and sort flow will bring issues since i need the date property inside the msg for the sort. Also can not just drop those two msgs from the sequence. That would prevent the split node to rebuild them aftwerwards. I have a solution, which is simple, but want to test it better.

0 Likes

#15

Thanks Andrei. I will do some research on object destructuring to make sure I fully understand it!

On the sorting problem do you have any suggestions?

Also, it might be good to give you a perspective on where I am going with this ultimately as you might be able to help me avoid potential stumbling blocks.

The final (for now) product is to build a chatbot which can answer various questions based on my gpx history. For example "when last was I at xyz place?" or "How much did I run last month?". I have been playing around with the API.AI chatbot from google and have it working with a python programme but I thought exploring node red would be a better idea for managing the flows ultimately.

For now I have around 6 years of data with a file (like the one supplied earlier) which I am trying to get into a structure which can be saved into a Influxdb database and then use the queries from that to drive the chatbot intents (I am also hoping that the google nodes might have an easy implementation for that with API.AI

Anyway for now my next steps are to sort the data and then get that flowing into Influx db. There is also 1 additional "column" I need to add to the data which is a calculation of distance between the lon and lat points in the file. This can be done using what is known as the haversine formula which can be done using a function node (I already know the javascript code for doing that)...but I am not sure whether to do that before pubslihing to influxdb or after.

You thought on this would be welcomed.

0 Likes

#16

Hi Gavin, there are tons of tutorials on JavaScript destructuring in the web. I would suggest trying this one first. I find it comprehensive, understandable and direct to the point. Furthermore, it is written by a known authority in the field.

I find your use case very interesting and compelling and feel that Node-RED will make your life easier.

The sorting issue is "sorted out" (I have it working) but I want to try two other possibilities before sharing the code with you in the hope I will choose the most appropriate approach (to avoid confusing you like may be the case with the data extraction).

In regards to calculating and storing the distance between points of a track. My perspective is that each line saved in the database is (should be) an individual piece of data. Storing this information along the line is like creating a link between those lines in the sense that the stored distance will not be meaningful without knowing to which pair of lines it is associated. In a first glance, it makes more sense calculating the distance on demand, after reading the points back from influxDB.

I am preparing my "home back" trip and should have a look at the flow in a couple of days.

0 Likes

#17

Hi Gavin,

Finally, I could have a look at the flow again. Honestly, the most difficult part for me is to make sense of the data. After playing with the example file you provided my understanding is that a waypoint is a different set of data when compared with a tracking. It is true that they share a lot in common but there is a reason the GPX format defines different structures for each one. Said that I don´t get why you want (or need) to come up with a single resulting file. It makes sense for me (but I may be possibly wrong or missing something) to generate two separate files: one for waypoints and another one for tracks. There is a lot of information that is specific to each of them. I came up with a (simple) flow that will handle the gpx file and will generate two files. The data is extracted in many different ways to be flexible to modify the output for your needs. Hopefully, data extraction is now more robust. I added the distance calculation for each set of points in each track segment. For that purpose, I installed the NPM module haversine-distance. I share the flow. If you have specific questions on how to adjust the output format to your need please contact me in private in the forum.

Flow:

[{"id":"3115eb81.6c01f4","type":"tab","label":"GPX  Parse - V2","disabled":false,"info":"Source:https://discourse.nodered.org/t/parse-gpx-file-and-extract-fields-from-javascript-object/7151/6\n\n"},{"id":"891d3776.11d948","type":"inject","z":"3115eb81.6c01f4","name":"Go","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":150,"y":160,"wires":[["f7def639.4fd288"]]},{"id":"e52422de.14b85","type":"change","z":"3115eb81.6c01f4","name":"Waypoint","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.gpx.wpt","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":180,"y":240,"wires":[["8dae91c3.a1b6"]]},{"id":"8dae91c3.a1b6","type":"split","z":"3115eb81.6c01f4","name":"Split0","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"topic","x":330,"y":240,"wires":[["ec1e9709.3c0248"]]},{"id":"ec1e9709.3c0248","type":"function","z":"3115eb81.6c01f4","name":"Format  waypoint","func":"try {\n    msg.pay = msg.payload;\n}\n\ncatch (err) {\n    msg.timesort = \"added\";\n    return msg;\n}\n\n\nmsg.timesort = msg.pay.time[0];\n\nlet {\n    lat, \n    lon} = msg.pay.$;\n\n\n\nlet {\n    time,\n    ele,\n    name,\n    src,\n    magvar,\n    geoidheight,\n    cmt,\n    desc,\n    sym,\n    type,\n    fix,\n    sat,\n    hdop,\n    vdop,\n    pdop,\n    ageofdgpsdata,\n    dgpsid,\n    extensions} = msg.pay;\n\ntry {\n    //let link = msg.pay.link[0].$.href;\n    msg.link = msg.pay.link[0].$.href;\n}\n\ncatch (error) {\n    msg.link = \"empty link\";\n}\n\n\n// Select data to be added in the output string that will be saves to influxDB\n\nmsg.output = `Date: ${time}, Lat: ${lat}, Lon: ${lon}, Ele: ${ele} Name: ${name}, Source: ${src} Link: ${msg.link}`;\n//alternatively msg.out = [time, lat, lon, ele];\n\nnode.status({text:msg.parts.index});\nreturn msg;","outputs":1,"noerr":0,"x":480,"y":240,"wires":[["ee09f536.7c47a8"]]},{"id":"3970ccb2.488464","type":"change","z":"3115eb81.6c01f4","name":"Track","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.gpx.trk","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":190,"y":320,"wires":[["d78a0205.f0dc"]]},{"id":"d78a0205.f0dc","type":"split","z":"3115eb81.6c01f4","name":"Split1","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"topic","x":330,"y":320,"wires":[["607fcf1d.aa6df"]]},{"id":"607fcf1d.aa6df","type":"function","z":"3115eb81.6c01f4","name":"Format track","func":"var haversine = global.get('haversine_distance');\n\nfunction distance(arr) {\n    let size = arr.length;\n    let d = 0;\n    for (let c=0; c < size-1; c++) {\n    d = d + haversine(arr[c], arr[c+1]);\n    }\nreturn d;\n}\n\n\ntry {\n    msg.seg   = msg.payload.trkseg[0];\n    msg.trkpt = msg.payload.trkseg[0].trkpt[0];\n}\n\n\ncatch (err) {\n    msg.timesort = \"added\";\n    //node.warn(msg);\n    return msg;\n}\n\n// trkseg may be empty - tag <trkseg></trkseg) or self closing <trkseg/>\n// results trkseg = [\"\"]; which is an array with oen element and content is an empty string\n// such case throw an error for the first line, not possible pay = \"\".trkpt[0]\n// the catch statement will inject a fake time into the msg part, so the sort node will not get lost\n    \nmsg.timesort = msg.trkpt.time[0];\n\n/*\nlet {\n    lat, \n    lon} = msg.trkpt.$;\n    \nlet {\n    time,\n    ele \n} = msg.trkpt;\n*/\n\n\nlet {\n    name,\n    type,\n    cmt,\n    desc,\n    src,\n    link,\n    number,\n    extensions} = msg.payload;\n    \nmsg.segsize = msg.seg.trkpt.length;\nmsg.segdata = msg.seg.trkpt.map(ob => ({\"lat\": ob.$.lat, \"lon\":ob.$.lon, \"ele\":ob.ele[0], \"time\": ob.time[0]}));\nmsg.distance = distance(msg.segdata);\n\n// Select data to be added in the output string that will be saves to influxDB\nmsg.output = `Date: ${msg.timesort}, Segment: ${msg.parts.index}, ${msg.segsize} Points: , Type: ${type}, Distance: ${msg.distance}, Name: ${name}, Cmt: ${cmt}, Desc: ${desc}, Src: ${src}, Link: ${link}, Number: ${number}, Extensions: ${extensions}`;\n\nnode.status({text:msg.parts.index});\nreturn msg;\n","outputs":1,"noerr":0,"x":490,"y":320,"wires":[["1759ed0e.91ada3"]]},{"id":"8b02947a.630268","type":"file","z":"3115eb81.6c01f4","name":"Save track.csv","filename":"C:/users/OCM/.node-red/static/nrfiles/track.csv","appendNewline":true,"createDir":false,"overwriteFile":"false","x":1060,"y":340,"wires":[[]]},{"id":"f7def639.4fd288","type":"file in","z":"3115eb81.6c01f4","name":"Read gps-4.gpx","filename":"C:/users/OCM/.node-red/static/nrfiles/gps-4.gpx","format":"utf8","chunk":false,"sendError":false,"x":360,"y":160,"wires":[["59f65c4a.fa0a54"]]},{"id":"59f65c4a.fa0a54","type":"xml","z":"3115eb81.6c01f4","name":"","property":"payload","attr":"","chr":"","x":530,"y":160,"wires":[["7c4d356.17ac0cc"]]},{"id":"bc7e7224.f0a03","type":"comment","z":"3115eb81.6c01f4","name":"Parse GPX file format","info":"","x":140,"y":80,"wires":[]},{"id":"ba723cc0.3a4e3","type":"comment","z":"3115eb81.6c01f4","name":"waypoint","info":"<...\nlat=\"latitudeType [1] ?\"\nlon=\"longitudeType [1] ?\"> \n<ele> xsd:decimal </ele> [0..1] ?\n<time> xsd:dateTime </time> [0..1] ?\n<magvar> degreesType </magvar> [0..1] ?\n<geoidheight> xsd:decimal </geoidheight> [0..1] ?\n<name> xsd:string </name> [0..1] ?\n<cmt> xsd:string </cmt> [0..1] ?\n<desc> xsd:string </desc> [0..1] ?\n<src> xsd:string </src> [0..1] ?\n<link> linkType </link> [0..*] ?\n<sym> xsd:string </sym> [0..1] ?\n<type> xsd:string </type> [0..1] ?\n<fix> fixType </fix> [0..1] ?\n<sat> xsd:nonNegativeInteger </sat> [0..1] ?\n<hdop> xsd:decimal </hdop> [0..1] ?\n<vdop> xsd:decimal </vdop> [0..1] ?\n<pdop> xsd:decimal </pdop> [0..1] ?\n<ageofdgpsdata> xsd:decimal </ageofdgpsdata> [0..1] ?\n<dgpsid> dgpsStationType </dgpsid> [0..1] ?\n<extensions> extensionsType </extensions> [0..1] ?\n</...>","x":200,"y":460,"wires":[]},{"id":"dfb309b3.9ed4f8","type":"comment","z":"3115eb81.6c01f4","name":"trk","info":"<...> \n<name> xsd:string </name> [0..1] ?\n<cmt> xsd:string </cmt> [0..1] ?\n<desc> xsd:string </desc> [0..1] ?\n<src> xsd:string </src> [0..1] ?\n<link> linkType </link> [0..*] ?\n<number> xsd:nonNegativeInteger </number> [0..1] ?\n<type> xsd:string </type> [0..1] ?\n<extensions> extensionsType </extensions> [0..1] ?\n<trkseg> trksegType </trkseg> [0..*] ?\n</...>","x":350,"y":460,"wires":[]},{"id":"72f8342e.ff6fac","type":"link in","z":"3115eb81.6c01f4","name":"","links":["7c4d356.17ac0cc"],"x":55,"y":320,"wires":[["3970ccb2.488464"]]},{"id":"7c4d356.17ac0cc","type":"link out","z":"3115eb81.6c01f4","name":"","links":["72f8342e.ff6fac","baa07b7c.d62748"],"x":655,"y":160,"wires":[]},{"id":"ee09f536.7c47a8","type":"sort","z":"3115eb81.6c01f4","name":"","order":"ascending","as_num":false,"target":"","targetType":"seq","msgKey":"","msgKeyType":"elem","seqKey":"timesort","seqKeyType":"msg","x":650,"y":240,"wires":[["237e0f1d.688b8"]]},{"id":"1759ed0e.91ada3","type":"sort","z":"3115eb81.6c01f4","name":"","order":"ascending","as_num":false,"target":"","targetType":"seq","msgKey":"","msgKeyType":"elem","seqKey":"timesort","seqKeyType":"msg","x":650,"y":320,"wires":[["56fa664e.0368c8"]]},{"id":"fe7de72d.435cd8","type":"catch","z":"3115eb81.6c01f4","name":"","scope":null,"x":200,"y":400,"wires":[["20df0f94.d6032"]]},{"id":"20df0f94.d6032","type":"debug","z":"3115eb81.6c01f4","name":"","active":true,"tosidebar":true,"console":true,"tostatus":false,"complete":"true","targetType":"full","x":340,"y":400,"wires":[]},{"id":"10d6039d.d832ec","type":"switch","z":"3115eb81.6c01f4","name":"","property":"payload","propertyType":"msg","rules":[{"t":"istype","v":"undefined","vt":"undefined"},{"t":"else"}],"checkall":"true","repair":false,"outputs":2,"x":910,"y":320,"wires":[[],["8b02947a.630268"]]},{"id":"100d284d.a115a8","type":"switch","z":"3115eb81.6c01f4","name":"","property":"output","propertyType":"msg","rules":[{"t":"istype","v":"undefined","vt":"undefined"},{"t":"else"}],"checkall":"false","repair":false,"outputs":2,"x":910,"y":240,"wires":[[],["fcf6b70f.e257d8"]]},{"id":"56fa664e.0368c8","type":"change","z":"3115eb81.6c01f4","name":"Output","rules":[{"t":"set","p":"payload","pt":"msg","to":"output","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":770,"y":320,"wires":[["10d6039d.d832ec"]]},{"id":"baa07b7c.d62748","type":"link in","z":"3115eb81.6c01f4","name":"","links":["7c4d356.17ac0cc"],"x":55,"y":240,"wires":[["e52422de.14b85"]]},{"id":"237e0f1d.688b8","type":"change","z":"3115eb81.6c01f4","name":"Output","rules":[{"t":"set","p":"payload","pt":"msg","to":"output","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":770,"y":240,"wires":[["100d284d.a115a8"]]},{"id":"fcf6b70f.e257d8","type":"file","z":"3115eb81.6c01f4","name":"Save waypoint.csv","filename":"C:/users/OCM/.node-red/static/nrfiles/waypoint.csv","appendNewline":true,"createDir":false,"overwriteFile":"false","x":1070,"y":260,"wires":[[]]}]
0 Likes