Hi there,
For reasons that go beyond the scope of this forum, I need to send large (> 20MB) files over Node-RED http-in node. So I created a simple flow:
The problem with this flow is that the read file node will create a single message with with all the file data. So a 20MB mp3 might just work but 180MB mp4 files won't - basically Node-RED freezes for a while and since this can cause users to be impatient, more requests are created as users access other content.
I began to wonder whether HTTP has a streaming functionality. I discovered yes, the chunked feature of HTTP/1.1 provides streaming support. I also found a forum question about using the chunked response with http response node - that got no traction.
So I began to think about how to implement this. Turns out read file has a "many small messages as chunks of data" option:
Chunking of file data was solved, now how to set the http header. Well that can be done in the http response node:
So the flow became this:
[{"id":"9447e2ddf16c5275","type":"http in","z":"543929cb2e9c4087","name":"","url":"/content2/:path","method":"get","upload":false,"swaggerDoc":"","x":549,"y":1123.5,"wires":[["254bfb6d6552d420"]]},{"id":"254bfb6d6552d420","type":"function","z":"543929cb2e9c4087","name":"set filename","func":"msg.filename = \"/data/content/\" + msg.req.params.path;\n\nreturn msg;","outputs":1,"timeout":"","noerr":0,"initialize":"","finalize":"","libs":[],"x":754,"y":1123.5,"wires":[["3fc061c241951ddf"]]},{"id":"3fc061c241951ddf","type":"file in","z":"543929cb2e9c4087","name":"","filename":"filename","filenameType":"msg","format":"stream","chunk":false,"sendError":false,"encoding":"none","allProps":true,"x":929,"y":1123.5,"wires":[["90d68e8d16222d2e"]]},{"id":"90d68e8d16222d2e","type":"http response","z":"543929cb2e9c4087","name":"","statusCode":"","headers":{"Transfer-Encoding":"chunked"},"x":1084,"y":1123.5,"wires":[]}]
(same image as above)
Turns out this does not work since the http response node closes the connection after sending the first chunk. Hm, damn I thought and did some soul searching.
Stackoverflow told me that it is down to using msg.res._res.send(...) instead of msg.res._res.write(...) to send data. It happens that express (the underlying HTTP framework) supports chunking simply by using write instead of send - in fact, it will automagically set the Transfer-Encoding header if write is used.
So I created my own function node to implement this behaviour:
[{"id":"9447e2ddf16c5275","type":"http in","z":"543929cb2e9c4087","name":"","url":"/content2/:path","method":"get","upload":false,"swaggerDoc":"","x":549,"y":1123.25,"wires":[["254bfb6d6552d420"]]},{"id":"254bfb6d6552d420","type":"function","z":"543929cb2e9c4087","name":"set filename","func":"msg.filename = \"/data/content/\" + msg.req.params.path;\n\nreturn msg;","outputs":1,"timeout":"","noerr":0,"initialize":"","finalize":"","libs":[],"x":759.6666666666666,"y":1123.25,"wires":[["3fc061c241951ddf"]]},{"id":"3fc061c241951ddf","type":"file in","z":"543929cb2e9c4087","name":"","filename":"filename","filenameType":"msg","format":"stream","chunk":false,"sendError":false,"encoding":"none","allProps":true,"x":940.3333333333333,"y":1123.25,"wires":[["162e98f2fba777a7"]]},{"id":"162e98f2fba777a7","type":"function","z":"543929cb2e9c4087","name":"send chunk","func":"msg.res._res.write(msg.payload)\n\n// the last buffer contains an count value in the parts hash.\n// if the count is set, then send an end response to the client.\nif ( \"count\" in msg.parts ) {\n msg.res._res.end()\n}\n\nreturn msg;","outputs":1,"timeout":"","noerr":0,"initialize":"","finalize":"","libs":[],"x":1121,"y":1123.25,"wires":[[]]}]
And that worked! So now my data is chunked and all clients are far more responsive (with content that is streaming complaint - i.e. mp3 and mp4 for example). This makes no difference if the client must receive all data before handling that data.
One thing that surprised me was the simplicity of the send chunk function node:
msg.res._res.write(msg.payload)
if ( "count" in msg.parts ) {
msg.res._res.end()
}
That's it. But this makes a big assumption: that messages are received in order - the http chunk transfer assumes correct order - there is no way to give a chunk an index in the HTTP/1.1 specs. The chunks are glued together as they are received.
This might well be an issue under load. So if the wire between read file and send chunk isn't sequential, then chunks are sent out of order and the data is corrupted on the client side.
This could be avoid by having something that checks the parts and ensures that messages are passed on in order, something like a order guarantee node - that would buffer only those messages that arrive out of order. The node would maintain the current parts number, i.e., last part sent was X therefore the next part to be sent will be X+1, all other messages are buffered until X+1 arrives - is there something like that?
