Filtering API Results

[dandspach]

Any assistance would be appreicated in getting this to work...my apologies if the post is not well formatted, I don't do a lot of forum posting

I have an HTTP API request that is working fine. I see all of the data which results in multiple arrays with multiple objects. However, after banging my head against the desk for several days, I'm reaching out for help. I want to do several things to "filter" the data before I send it to WLED (which I am quite capable of doing), but I need help with the filtering of the data first...

1. Filter all resulting HTTP API data to only result in 'spotTime' is within the last 10 minutes.
2. 'locationDesc' begins with "US-"
3. Unique concatinated values ('mode' + 'locationDesc' [state only]). Example: SSB-FL
   In the event that locationDesc has multiple states, then just the first is fine.

I am not planning on taking this into a dashboard, but rather sending to some WLED nodes to drive leds.

Here is my code:

[
{
"id": "09cc162097157e73",
"type": "tab",
"label": "Spots",
"disabled": false,
"info": "",
"env":
},
{
"id": "4b76beb994c633eb",
"type": "inject",
"z": "09cc162097157e73",
"name": "",
"props": [
{
"p": "payload"
},
{
"p": "topic",
"vt": "str"
}
],
"repeat": "",
"crontab": "",
"once": false,
"onceDelay": 0.1,
"topic": "",
"payload": "",
"payloadType": "date",
"x": 120,
"y": 60,
"wires": [
[
"c9bea5d026895da8"
]
]
},
{
"id": "c9bea5d026895da8",
"type": "http request",
"z": "09cc162097157e73",
"name": "spot request",
"method": "GET",
"ret": "obj",
"paytoqs": "ignore",
"url": "https://api.pota.app/spot/activator",
"tls": "",
"persist": false,
"proxy": "",
"insecureHTTPParser": false,
"authType": "",
"senderr": false,
"headers": ,
"x": 290,
"y": 60,
"wires": [
[
"3476258e1dfaf73f"
]
]
},
{
"id": "3476258e1dfaf73f",
"type": "debug",
"z": "09cc162097157e73",
"name": "debug",
"active": true,
"tosidebar": true,
"console": false,
"tostatus": false,
"complete": "payload",
"targetType": "msg",
"statusVal": "",
"statusType": "auto",
"x": 450,
"y": 60,
"wires":
}
]

[shrickus]

Could you post some sample API data for us to use in testing?

If you use 3-backticks around your data, it will be easier for us to copy it without html formatting getting in the way...

[UnborN]

Hello .. you can use the filter function to filter out the data you need and then map to restructure the data

In the flow below i used the moment js library that helps with the time calculations

In the example, as a test, i used 140 minutes because at the moment there are no data from the API that are within 10 min .. you can change that in code.

[{"id":"4b76beb994c633eb","type":"inject","z":"09cc162097157e73","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":120,"y":60,"wires":[["c9bea5d026895da8"]]},{"id":"c9bea5d026895da8","type":"http request","z":"09cc162097157e73","name":"spot request","method":"GET","ret":"obj","paytoqs":"ignore","url":"https://api.pota.app/spot/activator","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[],"x":290,"y":60,"wires":[["666e30c8b35f79e1","2efa07a436c55af3"]]},{"id":"3476258e1dfaf73f","type":"debug","z":"09cc162097157e73","name":"debug","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":670,"y":60,"wires":[]},{"id":"666e30c8b35f79e1","type":"function","z":"09cc162097157e73","name":"filter","func":"\n// filter less than 10 min  - 10*60*1000 ms\nlet data = msg.payload.filter(el => moment() - moment(el.spotTime) < (140 * 60 * 1000))\n\n// filter US-\ndata = data.filter(el => el.locationDesc.startsWith(\"US-\"))\n\ndata = data.map(el => { return { ...el, \"unique\": el.mode + \"-\" + el.locationDesc.split(\"-\")[1] } })\n\nmsg.payload = data\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[{"var":"moment","module":"moment"}],"x":470,"y":60,"wires":[["3476258e1dfaf73f"]]},{"id":"2efa07a436c55af3","type":"debug","z":"09cc162097157e73","name":"debug","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":470,"y":160,"wires":[]}]

[dandspach]

Ok, well that makes some sense. However, still on mystery to me is the 'filtering' of data. How does one get rid of the other fields (spotID, activator, frequency, etc...) in the Object?

I really need to get the data to the point where I can assign a 1 or 0 to the final WLED node.

For example:
The data would be filtered down to a single unique value:

At this point, I have all the Spots from each mode and state combination:
FT8-FL
FT8-CA
FT8-GA
FT8-FL

But I desire to have only the results as a unique dataset. This would allow me to do what I think I need to turn on a unique led for each mode-state combination.
FT8-FL
FT8-CA
FT8-GA

Thank you so much for the help so far. I'm working on a STEM project for schoolkids and want to demonstrate the power of data and how it can be literally mapped to the physical world.

[UnborN]

Some modification

[{"id":"4b76beb994c633eb","type":"inject","z":"09cc162097157e73","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":120,"y":60,"wires":[["c9bea5d026895da8"]]},{"id":"c9bea5d026895da8","type":"http request","z":"09cc162097157e73","name":"spot request","method":"GET","ret":"obj","paytoqs":"ignore","url":"https://api.pota.app/spot/activator","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[],"x":290,"y":60,"wires":[["666e30c8b35f79e1","2efa07a436c55af3"]]},{"id":"3476258e1dfaf73f","type":"debug","z":"09cc162097157e73","name":"debug","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":670,"y":60,"wires":[]},{"id":"666e30c8b35f79e1","type":"function","z":"09cc162097157e73","name":"filter","func":"\n// filter less than 10 min  - 10*60*1000 ms\nlet data = msg.payload.filter(el => moment() - moment(el.spotTime) < (140 * 60 * 1000))\n\n// filter US-\ndata = data.filter(el => el.locationDesc.startsWith(\"US-\"))\n\n\nlet unique = data.filter((value, index, self) =>\n  index === self.findIndex( t => ( t.locationDesc === value.locationDesc ))\n)\n\ndata = unique.map(el => { return { \"location\": el.mode + \"-\" + el.locationDesc.split(\"-\")[1] } })\n\nmsg.payload = data\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[{"var":"moment","module":"moment"}],"x":470,"y":60,"wires":[["3476258e1dfaf73f"]]},{"id":"2efa07a436c55af3","type":"debug","z":"09cc162097157e73","name":"debug","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":470,"y":160,"wires":[]}]

[dandspach]

Wow! Simply amazing. I can't thank you enough...that's it!

[dandspach]

Thank you so much. I've made awesome progress. However, one thing is throwing the final results off. For some reason, and I believe I know why, I am getting results that look like this:

location: "FT8-AL,US" or "CW-AL,US" or "SSB-AL,US"

Everything after the state abbreviation is causing issues with my led assignment.

Is there a way to limit the data returned to exclude anything after a comma using the filter code recommended by previous post above?

[UnborN]

@dandspach

a way to limit the data returned to exclude anything after a comma

yes .. i see the problem where in the data there are additional locations

we can handle it by doing an additional comma .split(",")[0] and taking only the first part
before the dash - split

code change (needs testing) :

...
data = unique.map(el => { return { "location": el.mode + "-" + el.locationDesc.split(",")[0].split("-")[1] } })
...

[dandspach]

That works.

One last question, then I'll leave you alone, I promise...

Regarding the time code filter.... Does it matter that the 'spotTime' is UTC? When I change to '1' or '1000' it doesn't seem to make a difference in the results. There is also an 'expire' field, which I'm assuming is in seconds and seems to count down to 0. Therefore, perhaps a better way may be to filter where 'expire' > [some amount of time]. Not sure...thoughts?

I want to say 'thank you again' for all of the assistance. I'm plowing through node red and realizing the power it has.

Below is a picture (taken yesterday-of the neopixels that are mapped to the various modes and states. I wanted to share my overall progress and let you know how happy this makes me!

Node red builds a .py file which drives the leds and the whole flow updates every 10 minutes. Really amazing!

[UnborN]

@dandspach

Does it matter that the 'spotTime' is UTC? When I change to '1' or '1000' it doesn't seem to make a difference in the results.

yes you are right .. it matters .. from what i read in the moment js library's docs

By default, moment parses and displays in local time.

so if the spotTime is in UTC then we need to modify the code to take that in consideration and do the comparison in UTC

possibly like this ? :

let data = msg.payload.filter(el => moment.utc() - moment.utc(el.spotTime) < (10 * 60 * 1000))

@dandspach

There is also an 'expire' field

maybe we can use that but i dont know what that field represents .. is there any online docs for the API ?

[dandspach]

You know, the code you sent fixed it all. Thank you again. Can I buy you a cup of coffee or something for your assistance?

[UnborN]

Im glad it worked ..
learning new things is enough as a reward .. that's what makes this forum great