HTTP request with variable date in address

General
1

Hello,

I'd like to send consecutive JSON/http requests and write the data to separate tabs on a Google Docs spreadsheet.

The historic data is stored on a server as day by day entries - I think in JSON format - and each one can be accessed by typing in the date it was generated (e.g. https://website.net/xxx-2020-09-19).

The resulting webpage is a list of fields which should map to columns in the Google Sheet e.g: {"min":24,"hr":1,"dow":"Sat","dom":19,"mon":12,"yr":2020,"imp":8880,"gen":600,"v1":2393,"frq":4988,"pect1":8880,"nect2":600,"hsk":300}

From what I've seen, I'm hoping it should be possible to automate the http request and just alter the final digits of the address as consecutive dates, but I'm struggling with how to do this.

I'm completely new to Node-RED but have done a little Dynamo and Grasshopper scripting so I get the concept of node-based programming.

Can anyone point to the best node/s to start with? I've seen BigTimer but this seems to be for future events unless I'm mistaken.

I'm running Node-RED on a Pi Zero WH if that makes any difference. Thanks in advance.

2

You can send the url to use to the http-request node in a msg so just format the url you need in a change node and pass it to the request node.

If you want to automate the date selection, you can use JSONata with the $now (to get the current date/time) and $moment (to format the output). You can use JSONata in the change node so you can do it all in 1.

Here is a simple example flow that returns the url using today's date.

[{"id":"7d825934.96cb18","type":"inject","z":"4a8d31b6.dfd47","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":150,"y":140,"wires":[["3382e9a5.ffb656"]]},{"id":"9c995c98.a006e","type":"debug","z":"4a8d31b6.dfd47","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"url","targetType":"msg","statusVal":"","statusType":"auto","x":630,"y":140,"wires":[]},{"id":"3382e9a5.ffb656","type":"change","z":"4a8d31b6.dfd47","name":"","rules":[{"t":"set","p":"url","pt":"msg","to":"\" https://website.net/xxx-\" & $moment().format(\"yyyy-MM-DD\")\t","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":300,"y":140,"wires":[["9c995c98.a006e"]]}]
3

@TotallyInformation thanks for the quick response. That looks like something I can use going forward so that future dates are recorded.

Is there a way to tweak the $now parameter to iterate through daily requests for the last 18 months though?

Essentially I’m trying to compile a load of historic data first.

4

Certainly.

As usual with Node-RED (and programming in general), there are various ways. However, to cover such a wide time period is probably easiest to achieve using a function node rather than a change node.

That is because you will want to:

  1. Capture the current date
  2. Generate an array of the last 18m dates
  3. Loop through the array and for each entry send a msg that goes to the request node

Here is something that may help with the date calculations: Subtract days from a date in JavaScript - Stack Overflow

5

Thanks, I've done some more digging and found the following which almost seems to work, but is formatted to give YY-DD-MM not YY-MM-DD. Even when I switch out the order of certain parts of the body text it seems to keep generating the wrong format or at least the debug window stops at the wrong date. Perhaps it's the UTC part that dictates the order?

[{"id":"2c55ba92.f8f226","type":"tab","label":"Flow 2","disabled":false,"info":""},{"id":"cb020599.1c4d48","type":"inject","z":"2c55ba92.f8f226","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":360,"y":240,"wires":[["624dd7f1.8f0988"]]},{"id":"f84398fe.40bfb8","type":"debug","z":"2c55ba92.f8f226","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":950,"y":240,"wires":[]},{"id":"624dd7f1.8f0988","type":"function","z":"2c55ba92.f8f226","name":"loop through dates","func":"var start = new Date(\"01/03/2005\"); //MM DD YYYY\nvar end = new Date(\"02/03/2005\");\n\n\nvar loop = new Date(start);\nwhile(loop <= end){\n\n    var day = loop.getUTCDate();\n    var month = loop.getUTCMonth() + 1; //months from 1-12\n    var year = loop.getUTCFullYear();\n\n    var str = year + \"-\" + month + \"-\" + day;\n    msg.payload=str;\n    node.send(msg);\n\n   var newDate = loop.setDate(loop.getDate() + 1);\n   loop = new Date(newDate);\n}\nmsg.payload=\"done\";\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":610,"y":240,"wires":[["f84398fe.40bfb8"]]}]
6

Here is how I simplified it - not sure what was going wrong with yours.

let day = new Date("2005-03-01"); //YYYY-MM-DD
var end = new Date("2005-03-02");

while(day <= end){

    msg.payload=day.toISOString();  // change this line to get the URL you want
    node.send(msg);

    day = new Date( day.setDate(day.getDate() + 1) )
}

msg.payload="done";
return msg;

Avoid using anything other than ISO format dates when programming with date strings, they are a problem waiting to happen.

7

Thanks. That works much better. Can I just check the comment line where you mention 'change this line to get the URL you want' are you suggesting that I should alter the body text of this function node to include the static parts of the URL? I assumed that I'd use this function node to create a (variable) payload that is then appended to a standard HTTP request node containing the rest of the URL.

8

You can do it either way. I'd probably just build the whole thing in one place because it is then easier if you ever go back to change things. That might just be me though! Either way is fine, whatever suits you.

9

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.