How do I save a file uploaded from dashboard locally in Node-RED?

General
1

I have simple form in the dashboard that lets me select and upload a file.

<form action="/save-csv" id="convert" method="POST" enctype="multipart/form-data">
    <div class="form-group">
        <label style="font-weight: 500">
            Select CSV file:
        </label>
        <div class="custom-file">
            <input id="file-input" type="file" name="file" accept=".csv" onclick="this.value=null;" class="custom-file-input" />
            <label class="custom-file-label" for="file-input">- select file -</label>
        </div>
    </div>
    <div class="form-group" style="btn-orange">
        <input type="submit" class="md-raised md-button md-ink-ripple" value="Convert" style="width:100%" />
    </div>
</form>

[{"id":"b1ca907d.8b7da","type":"http in","z":"8e883159.0b4bc","name":"","url":"/save-csv","method":"post","upload":true,"swaggerDoc":"","x":220,"y":1440,"wires":[["19201004.c4496","2156a9b9.5d0d06"]]},{"id":"19201004.c4496","type":"http response","z":"8e883159.0b4bc","name":"","statusCode":"","headers":{},"x":810,"y":1440,"wires":[]},{"id":"2156a9b9.5d0d06","type":"debug","z":"8e883159.0b4bc","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":390,"y":1400,"wires":[]},{"id":"b2cc0eef.f39a6","type":"comment","z":"8e883159.0b4bc","name":"how do I save or use the file's contents here?","info":"","x":530,"y":1440,"wires":[]}]

How do I save the file uploaded using the form above locally in Node-RED?
I want it to be saved temporarily so I can feed its contents later into the flow using the file-in node.

Also, I am requested to do this using only the built-in nodes so I can't use custom nodes.
Initially, I was looking into extracting the file contents from the http request's multipart data but it seems I would need to use custom nodes to accomplish it.

I am quite new to this so I might be looking into this problem incorrectly. Any advice, suggestion, or even a completely different kind of solution is appreciated. Thank you in advance for your help.

2

Hi

Have you searched this forum or the flows library?

Also, do you really need to save to file if its temporary? Can you not not receive the file data & perform your operation in one go?

3

Hi Steve,

Thanks for looking into my problem.

Yes, I browsed the flows but unfortunately, I didn't find anything that could help.

I tried but I had trouble on how extracting the data from the http request.
Can you give me an idea on how to approach this?

Also, I have a use case where the CSV data would be super huge (150mb).
That's why I thought of saving the file temporarily so I can use the "a msg per line" output mode in the file-in node. Because if I leave it as "a single utf-8 string", Node-RED crashes due to memory issues.

4

That's pretty big but not huge by any standards these days (your machine likely has 150MB memory free to hold this in memory)

I would pursue this first. I don't think node-red should be crashing.

I definitely recommend you start a new thread with title like "node-red crashes when I process 150MB CSV file", post a decent description, a copy of the file (as attachment, sanitised if necessary) and a minimal copy of your flow pasted between three ticks...

```
flow pasted here
```

Also include the nodejs version
node -v the version of node-red (get that from menu top right) and info on the OS where node red runs.

I think the devs will be interested (or have an answer for you on this)

There are probably (at least) 2 flows that can help in here.


Moving on...
It would seem you have gotten the upload of file to flow working - have you tried sending payload to a file node to see if it writes a valid file?

5

Well if you have uploaded it, it is already taking 150MB of memory, as it doesn't upload a line at a time ...
But as long as it's there you could process it right away -

[{"id":"5b46fc1b.dda6a4","type":"http in","z":"5051cb93.cf0d34","name":"","url":"/save-csv","method":"post","upload":true,"swaggerDoc":"","x":100,"y":1040,"wires":[["c42f61e1.b1859","f8dd08e5.326d28","19a06732.623e39"]]},{"id":"c42f61e1.b1859","type":"debug","z":"5051cb93.cf0d34","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":270,"y":1000,"wires":[]},{"id":"f8dd08e5.326d28","type":"function","z":"5051cb93.cf0d34","name":"","func":"msg.payload = \"OK\\n\";\nreturn msg;","outputs":1,"noerr":0,"x":500,"y":1040,"wires":[["6fd35dd7.cb6d54"]]},{"id":"19a06732.623e39","type":"function","z":"5051cb93.cf0d34","name":"","func":"msg.payload = msg.req.files[0].buffer.toString();\nreturn msg;","outputs":1,"noerr":0,"x":190,"y":1160,"wires":[["bfa1a505.2a5148"]]},{"id":"6fd35dd7.cb6d54","type":"http response","z":"5051cb93.cf0d34","name":"","statusCode":"","headers":{},"x":690,"y":1040,"wires":[]},{"id":"bfa1a505.2a5148","type":"csv","z":"5051cb93.cf0d34","name":"","sep":",","hdrin":"","hdrout":"","multi":"one","ret":"\\n","temp":"","skip":"0","strings":true,"x":330,"y":1160,"wires":[["77840964.cbb958"]]},{"id":"77840964.cbb958","type":"debug","z":"5051cb93.cf0d34","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":470,"y":1160,"wires":[]}]
6

Now, this is just silly. :rofl:
I didn't know I can just convert the buffer to string...
I'm still new to this javascript madness. :sweat_smile:

Thank you very much for your help, dceejay and Steve-Mcl.

7

Well in general you can't - which is why it's left as a buffer - but in this case as we know it's text for a csv file then we can guess it is and convert it.

8

hey @eoliverio exactly in which dashboard node you have written this code?