Downloading a csv from a link in a website

General
1

Hi,
I would like to download a csv file from a website once per day and save it to the local node-red file system as a csv file. (renaming the file would be nice but not crucial)

I do not know which node to use. I have tried a few and looked at available nodes from the palette but I can't see whether there is one that will facilitate this or not.

The path looks like this...
https://domainname.com/folder1/folder2/filename.csv

http request node - I thought might work, but seemingly not
csv nodes seem to be for working with local files only
file nodes also seem to need the file to be local

In the palette I saw this node...
@prescient-devices/node-red-contrib-downloadfile but from the description I couldn't decide whether this would work.

If anyone has done this type of thing, which node did you use?
Thanks
Richard

2

That is the correct note to use. Show us what you did and show us what you got.

Note this will get the file data but it will not save it to file. For that you need a file node.

3

Steve-Mcl,
Thanks for the reply.

I just exported the flow as follows....

[{"id":"7a39666703e3c4c8","type":"tab","label":"download","disabled":false,"info":"","env":[]},{"id":"233a5a7359cffc63","type":"inject","z":"7a39666703e3c4c8","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":140,"y":60,"wires":[["259c67f6bcb369e0"]]},{"id":"7b2169ecd53e8409","type":"debug","z":"7a39666703e3c4c8","name":"debug 62","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":740,"y":60,"wires":[]},{"id":"259c67f6bcb369e0","type":"http request","z":"7a39666703e3c4c8","name":"download csv","method":"HEAD","ret":"bin","paytoqs":"ignore","url":"https://ark-funds.com/wp-content/uploads/funds-etf-csv/ARK_INNOVATION_ETF_ARKK_HOLDINGS.csv","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[],"x":340,"y":60,"wires":[["36e0c77761307b1e"]]},{"id":"36e0c77761307b1e","type":"file","z":"7a39666703e3c4c8","name":"save csv file","filename":"//home/pi/today.csv","filenameType":"str","appendNewline":true,"createDir":false,"overwriteFile":"false","encoding":"none","x":550,"y":60,"wires":[["7b2169ecd53e8409"]]}]

Sorry, I had not posted code before.
I actually have a file of the right name but it just has an error in there. "error 1020"
I have also tried the GET option. Response has a 403 error

4

In order to make code readable and usable it is necessary to surround your code with three backticks (also known as a left quote or backquote ```)

``` 
   code goes here 
```

You can edit and correct your post by clicking the pencil :pencil2: icon.

See this post for more details - How to share code or flow json

5

edited, cheers

6

403 means "forbidden" see here

The HTTP 403 Forbidden response status code indicates that the server understands the request but refuses to authorize it.

This status is similar to 401, but for the 403 Forbidden status code, re-authenticating makes no difference. The access is tied to the application logic, such as insufficient rights to a resource.

That said, it works for me (when using GET method not HEAD!)...

image
image1053×420 37.2 KB

[{"id":"233a5a7359cffc63","type":"inject","z":"7a39666703e3c4c8","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":140,"y":60,"wires":[["259c67f6bcb369e0"]]},{"id":"259c67f6bcb369e0","type":"http request","z":"7a39666703e3c4c8","name":"GET ARK_INNOVATION_ETF_ARKK_HOLDINGS.csv","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://ark-funds.com/wp-content/uploads/funds-etf-csv/ARK_INNOVATION_ETF_ARKK_HOLDINGS.csv","tls":"","persist":false,"proxy":"","authType":"","senderr":false,"headers":[],"credentials":{},"x":460,"y":60,"wires":[["36e0c77761307b1e","e2fe1217aa40215a"]]},{"id":"36e0c77761307b1e","type":"file","z":"7a39666703e3c4c8","name":"save csv file","filename":"c:/temp/test.csv","filenameType":"str","appendNewline":false,"createDir":false,"overwriteFile":"true","encoding":"none","x":790,"y":120,"wires":[[]]},{"id":"e2fe1217aa40215a","type":"debug","z":"7a39666703e3c4c8","name":"debug 156","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":790,"y":60,"wires":[]}]

What version of node-red and nodejs are you using?

  • enter node -v in command line to get nodejs version
  • see node-red menu for node-red version
  • how are you running node-red - in a container? (e.g. docker? as a Home Assistant plugin? other?)
7

Thanks so much for your help.
nodejs is 18.12.1

node-red is 3.0.2

It is on a Raspi 4 and the only thing it does is node red and postgresql - not docker container

I hit an outside api from node red regularly so it should not be a routing issue.

I will try your code snippet. I do have some other instances of node red I could try it on but thought my versions were reasonably up to date.
Cheers
Richard

8

Is there a proxy between the PI and internet?

9

Steve-Mcl
using your copied back flow...
I tried on another machine on our home network (non raspi) and same error.same deal, not docker but it is on a vm which also accesses the outside world.

I have a server in a datacentre similar configuration and same error.
It hardly seems fair that it worked for you - oh well, that's computers.

I'm not sure what to try next. I felt sure the datacentre one would work after it worked for you.
All my machines are linux, Ubuntu.

10

were they all nodejs 18?

I was using nodejs 16 on a windows box

11

Steve-Mcl,
I would have to check but I suspect so, yes.

Interesting outcome and annoying.
First time I have encountered such a problem. I do use the get for my api queries, so it is working for that functionality.

I should perhaps keep an eye on updates in the coming week/month. Maybe there is a gremlin floating around in there.

I will have to shelve that planned project for a bit I think.
Appreciate you taking the time to try for me and assisting.
Cheers
Rich

12

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.