File Upload through Uibuilder NodeRED Interface

[stefanoanzilotti]

Dear all,

First of all, a little introduction to my system.
I am using a docker container to run several services, one of them is obviously NodeRED.
here I use an amazing node-red-contrib-uibuilder to pull up my interface with HTML, CSS, and JS language (No Vue or React is used).

I go straight to the problem
I would like to upload, from this interface, a CSV / TXT file and store it in a server-side folder.

Wandering around the net I found some way like:

• node-red-contrib-http-multipart
• node-red-contrib-file-upload

right now I am able just to read the file name in the debug section of NodeRED, but I do not know how to access the entire file.

Here same more detail about my work (following the previous link):

• HTML PAGE:

IP:1880/uibuilder/import.html

• HTML CODE:

<form action="/upload" method="POST">
<input type="file" name="myFile" />
<input type="submit" />
</form>

• NR FLOW:

• NR FUNCTION:

var fields = msg.req.fields;
msg.fields = Object.keys(fields);
var myFile = fields["myFile"][0];
msg.localFilename = myFile.path
return msg;

• NR OUTPUT

Moreover, the HTML page keeps loading after the SUBMIT click.

Hope to have been clear.
Thank you in advance,

Stefano

[shrickus]

I see at least 2 things to check:

• the http in node for /upload needs to have the option "Accept File Uploads" checked
• and the /upload flow needs to have a http response node at the end

You may also want to set the html form element encoding to use multipart/form-data, but I think that's implied when using the file selector.

[TotallyInformation]

@stefanoanzilotti

<form action="/upload" method="POST">

Possibly try changing that to <form action="/upload" method="POST" enctype="multipart/form-data">

[TotallyInformation]

Interestingly, Socket.IO has a binary file transfer capability, I didn't know that. I'll add it to the roadmap as something to maybe enable in uibuilder so that you can send a file as a buffer.

[shrickus]

@stefanoanzilotti

Moreover, the HTML page keeps loading after the SUBMIT click.

Without an http response node at the end of your flow, the browser never gets an answer and keeps waiting until it times out (in 90 secs or so) with a 504 Gateway Timeout status (iirc)

You can set the response code in your http response node to be 204 (No Content) in order to keep from navigating to an empty page.

[stefanoanzilotti]

Thank you both guys @shrickus and @TotallyInformation.
The main problem was related to the missing feature multipart/form-data.
I still need to work to extract filename and path to correctly store it, but I can handle it.

Regarding the keep loading issue, as from @shrickus, I trayed to add an HTTP response node.
It works perfectly, but I would like to keep seeing the same page of the form. Do you have some more advice?

[TotallyInformation]

Been a while since I looked at that. But do you need to cancel the default action of the form?

[shrickus]

@stefanoanzilotti

I would like to keep seeing the same page of the form

Adding statusCode 204 in the http response node should cause the browser to stay on the current page...

Otherwise, you may have to write a submit function, that returns false to stop the browser from refreshing the page.

[stefanoanzilotti]

Actually, I did not think about it. Let's say it is not mandatory.
Moreover, guys, do you have some information about the security of this upload?

For the moment it seems perfect, but I can try to improve it in a while if the transfer is secure enough.

[shrickus]

@stefanoanzilotti

I still need to work to extract filename and path to correctly store it, but I can handle it.

Those fields are already parsed for you by node-red, and are available on msg.req.files, I believe. Change your debug node to show the "Complete msg object" and you will find what you need.

[TotallyInformation]

@shrickus

Adding statusCode 204 in the http response node should cause the browser to stay on the current page...

Nice!

@shrickus

msg.req.files

Correct - it returns a BUFFER.

This flow works but you will want to change the filename slightly since it currently dumps the file in the Node-RED install folder.

I can confirm that the actual page is NOT reloaded - that's good since it doesn't lose any live data on the page. However, you will want to have a script that empties the filename so that people don't keep uploading the same file.

[{"id":"6081b5d0ba078956","type":"http in","z":"bfec3af46a41235d","name":"","url":"/upload","method":"post","upload":true,"swaggerDoc":"","x":280,"y":380,"wires":[["54b563baa08e10b5","add8ee6a9e62361e"]]},{"id":"54b563baa08e10b5","type":"http response","z":"bfec3af46a41235d","name":"","statusCode":"204","headers":{},"x":800,"y":380,"wires":[]},{"id":"c86833183e63bf4c","type":"debug","z":"bfec3af46a41235d","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":890,"y":460,"wires":[]},{"id":"c6fed61b5941f570","type":"file","z":"bfec3af46a41235d","name":"","filename":"","appendNewline":false,"createDir":false,"overwriteFile":"true","encoding":"none","x":720,"y":460,"wires":[["c86833183e63bf4c"]]},{"id":"add8ee6a9e62361e","type":"change","z":"bfec3af46a41235d","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"req.files[0].buffer","tot":"msg"},{"t":"set","p":"filename","pt":"msg","to":"req.files[0].originalname","tot":"msg"},{"t":"set","p":"encoding","pt":"msg","to":"req.files[0].encoding","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":520,"y":460,"wires":[["c6fed61b5941f570"]]}]

Going to write this up on the uibuilder WIKI for future use.

@stefanoanzilotti

if the transfer is secure enough

The transfer is only secure if you make it so. You need to at least implement TLS encryption (e.g. use HTTPS not HTTP). Then you should also validate the upload data before blindly assuming that it is safe. For example, check the file mimetype matches the filename extension, block unsafe file types (or better still have an allow list of types), limit the filename length and characters, limit the buffer size.

You should also only be allowing updates from known and authenticated users.

You might also want to have checks of available storage before committing the file and some other process to clear down old files or limit space utilisation.

[TotallyInformation]

How to upload a file from the browser to Node RED ·...

[stefanoanzilotti]

Thank you @TotallyInformation this last guide has been really useful.

Now I notice a now issue due the following message:

[uibuilderfe:socket-disconnect] Reason: ping timeout

I also followed this POST, but I did not work for me.

[TotallyInformation]

So something is creating a delay between your client and the server. Most commonly, this is due to trying to pass really large data in a single msg.

One way to fix that would be to split your file in the client code if it is too large and then send each of the pieces and put them back together in node-red.

The other way requires you to mess with the socket.io server settings. Under uibuilder v4, there is no mechanism for that I'm afraid - unless you manually hack the appropriate uibuilder source file.

However, if you are prepared to run a beta version, you could install the GitHub vNext branch which is what will soon be uibuilder v5. In this version, you can supply settings for socket.io by adding the suitable properties to settings.js. It might be worth you creating a test instance of node-red with uibuilder vNext and trying it out to make sure it works.