Lookup JSON response in another JSON response

[g7kse]

I'm extremely new to all of this and have a small issue to do with JSON

I am trying to build up a small dashboard for a satellite project. Data is held in a couple of JSON URL's. The first API is one called Observations, this holds information relating to ground stations and the observations it has scheduled (https://network.satnogs.org/api/observations/?id=&ground_station=33&satellite__norad_cat_id=&transmitter=&vetted_status=&vetted_user=&start=&end= for example is my ground station).The second API is called db (as in database) where specific data about the satellite is held, for example transmitter data (https://db.satnogs.org/api/transmitters/). The two are linked by a unique id reference, below is the flow as it is. I have highlighted the transmitter as it is in the first API, the same data is called uuid in the second API

And here in the Database

The crux of the question is how do I go about looking up the response from the flow from the first instance in the second API and so allowing me to return more useful information such as it's description. I can then use the same principle to use the NORAD id to return a satellite name?

It all sounds fairly straightforward, i.e. look something up somewhere and cross reference it against some other information. Any help would be appreciated.

[Andrei]

Hi @g7kse,

I can imagine the following solution.

A flow to read the JSON from the transmitters API and store the data (as JSON) in a global context.

A second flow that will read the JSON data from the observations API and will split each observation in a separate message. A change node with a JSONata expression will use the transmitter property from the split message to look up the stored JSON. The full transmitter data will be added to the message as the property "payload.transmitter_full".

This way you will have data from both JSON APIs together (under msg.payload) making easier to manipulate (extract) the relevant data.

Sounds, complicated right ? Try running the flow and explore the result in the debug tab.

[{"id":"87eb59d5.c9fa88","type":"tab","label":"SatNOGS","disabled":false,"info":""},{"id":"ae8c24c1.db8728","type":"inject","z":"87eb59d5.c9fa88","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":140,"y":280,"wires":[["44ded748.9c4888"]]},{"id":"44ded748.9c4888","type":"http request","z":"87eb59d5.c9fa88","name":"","method":"GET","ret":"obj","paytoqs":false,"url":"https://network.satnogs.org/api/observations/?id=&ground_station=33&satellite__norad_cat_id=&transmitter=&vetted_status=&vetted_user=&start=&end=","tls":"","proxy":"","authType":"basic","x":310,"y":280,"wires":[["eff5d47d.bbca48"]]},{"id":"6f1ecae6.6cc1c4","type":"inject","z":"87eb59d5.c9fa88","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":140,"y":200,"wires":[["20a28e41.4b5d02"]]},{"id":"6f89bb3e.9eaa44","type":"debug","z":"87eb59d5.c9fa88","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":710,"y":200,"wires":[]},{"id":"20a28e41.4b5d02","type":"http request","z":"87eb59d5.c9fa88","name":"","method":"GET","ret":"obj","paytoqs":false,"url":"https://db.satnogs.org/api/transmitters/","tls":"","proxy":"","authType":"basic","x":300,"y":200,"wires":[["3f2641f1.bab68e"]]},{"id":"3f2641f1.bab68e","type":"change","z":"87eb59d5.c9fa88","name":"","rules":[{"t":"set","p":"transmitters","pt":"global","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":500,"y":200,"wires":[["6f89bb3e.9eaa44"]]},{"id":"eff5d47d.bbca48","type":"split","z":"87eb59d5.c9fa88","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":480,"y":280,"wires":[["fec5ee3f.3f139"]]},{"id":"fec5ee3f.3f139","type":"change","z":"87eb59d5.c9fa88","name":"","rules":[{"t":"set","p":"payload.transmitter_full","pt":"msg","to":"$globalContext(\"transmitters\")[uuid= $$.payload.transmitter]","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":690,"y":280,"wires":[["eb014cf.21eb7b"]]},{"id":"eb014cf.21eb7b","type":"debug","z":"87eb59d5.c9fa88","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":890,"y":280,"wires":[]}]

[g7kse]

Thanks @Andrei A really quick response. I think I follow you, I'll have a play a bit later and see where that takes me. I had myself really stuck with this and was thinking about tables, variables and all sorts but this looks easier

Alex

[g7kse]

This works really well for transmitters so thanks for that, it now makes it easy to set up a dshaboard and is exactly what I was after. I had a play with the satellite names but didn't manage to replicate this with the different source. Debug shows that the JSON data is coming in correctly but the information is not appended as before.

I've had a play but can't seem to spot the error. Once I have this sorted I will try and get both sources into a simple flow.

Really enjoying the Node-RED way of doing stuff, quite easy to follow after a bit of a play around

[{"id":"8f21245d.58fd7","type":"tab","label":"SatNOGS","disabled":false,"info":""},{"id":"947c3a88.05ef58","type":"inject","z":"8f21245d.58fd7","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":140,"y":280,"wires":[["e8f83218.c8b258"]]},{"id":"e8f83218.c8b258","type":"http request","z":"8f21245d.58fd7","name":"","method":"GET","ret":"obj","paytoqs":false,"url":"https://network.satnogs.org/api/observations/?id=&ground_station=33&satellite__norad_cat_id=&transmitter=&vetted_status=&vetted_user=&start=&end=","tls":"","proxy":"","authType":"basic","x":310,"y":280,"wires":[["81a8d789.6f4fa8"]]},{"id":"c3436597.ce4758","type":"inject","z":"8f21245d.58fd7","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":140,"y":200,"wires":[["540ea906.36a11"]]},{"id":"4d330d34.03a324","type":"debug","z":"8f21245d.58fd7","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":710,"y":200,"wires":[]},{"id":"81a8d789.6f4fa8","type":"split","z":"8f21245d.58fd7","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":480,"y":280,"wires":[["fbcc3f21.8aeea8"]]},{"id":"1b907627.e47952","type":"debug","z":"8f21245d.58fd7","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":890,"y":280,"wires":[]},{"id":"540ea906.36a11","type":"http request","z":"8f21245d.58fd7","name":"","method":"GET","ret":"obj","paytoqs":false,"url":"https://db.satnogs.org/api/satellites/","tls":"","proxy":"","authType":"basic","x":310,"y":200,"wires":[["b3f7a3a7.b0ac5"]]},{"id":"b3f7a3a7.b0ac5","type":"change","z":"8f21245d.58fd7","name":"","rules":[{"t":"set","p":"satellites","pt":"global","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":520,"y":200,"wires":[["4d330d34.03a324"]]},{"id":"fbcc3f21.8aeea8","type":"change","z":"8f21245d.58fd7","name":"","rules":[{"t":"set","p":"payload.satellite_full","pt":"msg","to":"$globalContext(\"satellites\")[norad_cat_id= $$.payload.satellites]","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":690,"y":280,"wires":[["1b907627.e47952"]]}]

[Andrei]

@g7kse

Debug shows that the JSON data is coming in correctly but the information is not appended as before.

I've had a play but can't seem to spot the error. Once I have this sorted I will try and get both sources into a simple flow.

You were very close. Looks like observations and satellites data structures use a coincident key name: norad_cat_id. Try using below config in the flow you created.

[g7kse]

Brilliant...Tnx again @Andrei. It had me stumped for a couple of hours trying this and that. Now for a swanky dashboard

[Andrei]

and this flow add both: transmitter and satellites to each observation.

[{"id":"d8d2b8b1.4b6988","type":"tab","label":"SatNOGS-3","disabled":false,"info":""},{"id":"ab23037b.cde82","type":"inject","z":"d8d2b8b1.4b6988","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":160,"y":280,"wires":[["99f77fd4.74041"]]},{"id":"99f77fd4.74041","type":"http request","z":"d8d2b8b1.4b6988","name":"","method":"GET","ret":"obj","paytoqs":false,"url":"https://network.satnogs.org/api/observations/?id=&ground_station=33&satellite__norad_cat_id=&transmitter=&vetted_status=&vetted_user=&start=&end=","tls":"","proxy":"","authType":"basic","x":330,"y":280,"wires":[["642bf6ce.2f0c38"]]},{"id":"75b54a8f.4ca984","type":"inject","z":"d8d2b8b1.4b6988","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":true,"onceDelay":"0.5","x":150,"y":200,"wires":[["92584431.b72438"]]},{"id":"77572136.b36d8","type":"debug","z":"d8d2b8b1.4b6988","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":730,"y":200,"wires":[]},{"id":"642bf6ce.2f0c38","type":"split","z":"d8d2b8b1.4b6988","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":500,"y":280,"wires":[["6756dd9e.2914e4"]]},{"id":"eed749fb.6857f8","type":"debug","z":"d8d2b8b1.4b6988","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":930,"y":280,"wires":[]},{"id":"92584431.b72438","type":"http request","z":"d8d2b8b1.4b6988","name":"","method":"GET","ret":"obj","paytoqs":false,"url":"https://db.satnogs.org/api/satellites/","tls":"","proxy":"","authType":"basic","x":330,"y":200,"wires":[["3bc6f88f.57cf48"]]},{"id":"3bc6f88f.57cf48","type":"change","z":"d8d2b8b1.4b6988","name":"","rules":[{"t":"set","p":"satellites","pt":"global","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":540,"y":200,"wires":[["77572136.b36d8"]]},{"id":"6756dd9e.2914e4","type":"change","z":"d8d2b8b1.4b6988","name":"","rules":[{"t":"set","p":"payload.satellite_full","pt":"msg","to":"$globalContext(\"satellites\")[norad_cat_id= $$.payload.norad_cat_id]","tot":"jsonata"},{"t":"set","p":"payload.transmitter_full","pt":"msg","to":"$globalContext(\"transmitters\")[uuid= $$.payload.transmitter]","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":670,"y":280,"wires":[["eed749fb.6857f8"]]},{"id":"737f1a11.320a14","type":"inject","z":"d8d2b8b1.4b6988","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":true,"onceDelay":0.1,"x":150,"y":160,"wires":[["d614c56e.3bc548"]]},{"id":"126302fb.bbc0dd","type":"debug","z":"d8d2b8b1.4b6988","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":730,"y":160,"wires":[]},{"id":"d614c56e.3bc548","type":"http request","z":"d8d2b8b1.4b6988","name":"","method":"GET","ret":"obj","paytoqs":false,"url":"https://db.satnogs.org/api/transmitters/","tls":"","proxy":"","authType":"basic","x":320,"y":160,"wires":[["731ca76f.62e938"]]},{"id":"731ca76f.62e938","type":"change","z":"d8d2b8b1.4b6988","name":"","rules":[{"t":"set","p":"transmitters","pt":"global","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":520,"y":160,"wires":[["126302fb.bbc0dd"]]}]

It is a pitty that the satellite image is not always available. That would make the dashboard awesome.

This project seems really nice. I feel tempted to build my own ground station. If I can use a discone antena that is lying around then it is missing only to find a RTL SDR dongle.

[g7kse]

It is a great project. I have had a ground station for a couple of years now using a NooELEC SMArt RTL-SDR. It cost around $20 from eBay. It is really just a $5 device with a TXCO and an sma coax connector in a metal box. The performance is what you would expect from such a device.

I can't see a discone giving you much though. Is it not vertically polarised? If you are really up for it then I have a spare turnstile antenna that I originally used. PM me your details and I'll get it sent. This will give you better results overhead. I changed it for a Wimo TA-1 about a year ago and that gives a slight reduction overhead but improved low angle reception.

A long time ago I built the 3d printed rotator but found the control side very complex and prone to failure. Recently I've moved it from the custom board to a Arduino Uno and CNCshield which is far easier and cheaper. I wrote the wiki page to show that it was much easier to do it that way and also it gives independence. (I can only post 2 links so search wiki.satnogs)

After all of that the static antenna, Nooelec SMArt and a turnstile gives great results. It can be 100% utilised without any danger of it breaking down. The UHF station is giving very poor results though.

[Andrei]

Hello @g7kse,

Indeed, I read somewhere that the discone antenna would not be a good choice for this case. Mine is an ICOM AH7000 (VHF-UHF , 25 ~ 1300 MHz, omnidirectional ). This antenna and an old Hoka code 3 decoder was everything left from the time I was active scanning RF. I will take some time to study the SatNOGs project before taking the decision to build something. In the meantime let me know if I can help with the dashboard challenge. What you plan to develop is something similar to https://dashboard.satnogs.org ?

[g7kse]

I was really aiming a bit lower (sort of matched my skills ). I was going to aim to re use a discarded kindle as an epaper display for the local groud station, perhaps build on that.

The SatNOGS dashboard is way more complex as ground stations have, up until quite recently, only provided demodulated data from the sdr. there has been a load of effort to be able to reliably decode that data and then provide telemetry dashboards for some of the newer cubesats. The data comes in loads of formats like CW (morse code), FSK, AFSK etc and it is quite complex.

As a project it is moving very quickly and t starting to become a useful tool. It is a bit like Piaware that is used for aircraft ADSB tracking but a whole load fewer stations at the moment

[thebaldgeek]

@g7kse I added this flow this morning.
https://flows.nodered.org/flow/a2fe839215c1b3ab77fe2b3c9fd5b659
Looking at your flow, I am not really sure what its returning, perhaps you could explain it a little more.
Also, did you continue building a dashboard? I'd love to see and share what you are up to.

[g7kse]

I did do a bit more but never really got it how I wanted it. The idea was to use the SatNOGS API to return the next scheduled pass and display it using the inbuilt web thingy on an old kindle and for that to get refreshed after each of the passes had been completed. I got stuck with the dashboard and kindle, just couldn't get it to work and thought I needed to do a bit of hacking with the kindle. In the end I got to re-writing this in rubbish python (As you can tell I am no coder) and used a scrolling display called a Scroll Phat HD with a RPi w The code and video sort of show it. Although I would still like for there to be an ePaper solution. I just ran out of time as the weather got better and I went outside a bit more

Have you got any thoughts?

I think perhaps it is the wrong API and it is making it hard work

[thebaldgeek]

Take a look at the Node-RED flow I posted. I does exactly what you describe.
It shows UTC time, the time of the next pass, how long in minutes, what the transmitter is, what frequency it is and so on.
I use an old iPad as my SatNOGS display.
As you know, with Node-RED it is trivial to make a dashboard and I think the Kindle has a web browser, so you should be up and running in a few minutes.

[dceejay]

Of course we do use a lot of recent JavaScript. So it may be that the browser on the kindle isn't up to date enough.

[g7kse]

Nice @thebaldgeek far better than mine. Although I get a "Output field is REQUIRED, currently blank, set to msg.payload" on the UTC node. Perhaps something to do with moment? Either way it has sorted itself out

[g7kse]

It struggled to even consistently get to a local ip address. Some more investigation should see if it is worth pursuing especially as a RPi zero w is so cheap

[thebaldgeek]

@g7kse

"Output field is REQUIRED, currently blank, set to msg.payload"

Here is a screen shot of my Moment config. Make sure yours matches and it should take care of that error;

Of course you will need to change your timezone to suit.

Regarding the Pi Zero, it might be struggling to run both the SatNOGS client AND Node-RED.
Also your wifi might be an issue. The Zero wifi is know to be a bit weak.

Let me know if you get the Kindle going. Like dceejay said, Node-RED uses pretty hard core JavaScript. I have an even older iPad and it will not show the even the Node-RED dashboard status bar.
Note that with the Eink, you may need to slow down the UTC refresh rate a lot.

[g7kse]

I got it going in the end. Just did it all by itself so no idea what the bother was. I wasn't going to run the SatNOGS client on the zero. I'm pretty sure it wouldn't manage. I was after a simple desktop solution which is pretty much there. I don't know if you've seen the Wose solution. It is another way of displaying the data but much more work

Might be nice to tidy up the date /time and frequency but other than that it is spot on. Nicely done.