Trivial I think compared to the amount of work ffmpeg is doing decoding the rtsp stream.
Of course Dave is always right, but like my wife always says: "Trust is good, but control is better...". So to gather some evidence, I have quickly debugged where the chunks (in the "data" event of the Exec-node) are coming from. Seems the chunks have indeed already a size of 65K when the come out of the socket:
So my highWaterMark level of 128K is never reached, and has no effect at all...
And for the non-believers among us, here is an extra explanation from the NodeJs team, that the chunk size is determined by the operating system.
...
replace the ffmpeg (or curl command in Bart's example) with:
ffmpeg -i /path/to/local/1080p.mp4 -f image2pipe pipe:1
The size of the output jpegs will depend on the bit rate of the videofile, but the imageOutput display will break up, flash, or otherwise misbehave when the jpeg frame exceeds the highwatermark.
If Bart's RIP is correct, too bad, as this would have been an extremely slick way to convert rtsp streams into MQTT buffers for frame by frame processing.
Yes but only the idea (to adapt the Exec-node to generate a single chunk per image) is RIP. But if we can use the Join or Split nodes - as Dave and Michael suggest - efficiently to concatenate the chunks into a single image, then we are back on track I assume. I have never used those nodes, so the others will be of more help to you from here on ...
I think there is a fundamental issue with trying to use join and split nodes on binary buffers as produced by ffmpeg piping frames to stdout.
There is a set of "magic bytes" at the start of standardized compressed file formats, jpeg for example generally start with 0xFF 0xD8 0xFF but there is nothing to signal the end of the file and as far as I know nothing in the jpeg compression standard that would prevent this string from occurring somewhere in the compressed data stream.
If the stream is a sequence of images there must be some way for the receiver to know when one image finishes and the next starts.
But can't we just do it like this: we collect data until we see the magic byte sequence. Then we now that the previous image is complete, so we can send an output image. And we start collecting the bytes of the next image. Think it is enough to have the start of the image, since that is also the end of the previous image ... I do something similar in my node-red-contrib-mutlipart-stream decoder, to find the boundary (i.e. separator) between jpegs in an mjpeg stream. Only disadvantage is that it will consume cpu to find the magic bytes in the buffer !!!
For what it's worth the Wikipedia article on jpeg says that
-
The start of an image is marked by 0xFF 0xD8
-
The end of an image is marked by 0xFF 0xD9
-
When 0xFF occurs in the data stream, the encoder inserts 0x00 after it so that it won't be mistaken for a marker
It might be worth trying to split on 0xFF 0xD9 with "Handle as a stream of messages" enabled. The 0xFF 0xD9 bytes would have to be added back to the message, but it should contain an entire image.
Hey Michael,
We will have to do things like that with caution. If we do it incorrectly, NodeJs will create a entirely new buffer that is long enough to store the original buffer + the new magic byte sequence. But that means NodeJs will copy the entire image bytes behind our backs, which is not what we want (for performance).
P.S. In my multipart stream decoder I had in the beginning lots of issues, when the magic bytes are divided across the boundary between two chunks. Part of the magic bytes are in at the end of chunk N and the rest of the magic bytes are at the start of the chunk N+1. Found a solution at last, but just something we have to think about...
I was just about to post about that. I suspect that Node-RED might have already duplicated buffers created by ffmpeg, so performance is certain to be an issue.
1 Like
Don't know it you can find byte sequences (even across chunk boundaries) with the Split or Join nodes? Anyway I developed last month an alpha version of a Buffer-node, which contains a.o. this kind of functionality. If anybody is interested I can publish next week a test version of it ...
I don't know. poking around on some jpeg images with GHEX "dump" program there are quite a few 0xFF characters incluiding some followed by 0xD8 and others followed by 0xD9 I'd be expecting false start and stop flags.
However looking at a few jpeg images produced by:
ffmpeg -f rtsp -i "rtsp://admin:xyzzy@192.168.2.164:554/cam/realmonitor?channel=4&subtype=0" -f image2 out_img-%03d.jpe
from my security DVR rtsp streams, it does look like 0xFFD8 always starts the file and 0xFFD9 always ends it. So perhaps it could work in this situation,
Good point, and I missed it. The split node will accept multi-character (and I assume multi-byte) delimiters, and the "Handle as a stream of messages" logic ought to put things right, but I will write a test flow to look at this.
1 Like
If you figure it out I will certainly test it in short order!
something like this maybe
[{"id":"513524c9.1e097c","type":"function","z":"64d407f5.647078","name":"","func":"var b = Buffer.concat([(context.buf || Buffer.from(\"\")), msg.payload]);\n\nvar i = b.indexOf(\"FFD9\", 0, \"hex\");\nif (i >= 0) {\n var frame = b.slice(0,i+1);\n context.buf = b.slice(i+2);\n node.send({payload:frame});\n}\nelse {\n context.buf = b;\n}\nreturn null;","outputs":1,"noerr":0,"x":600,"y":500,"wires":[["15aa84d7.ed77ab"]]}]
1 Like
Yes, I tested with multi-character delimiters, and it works ok. I think a chunk containing a partial delimiter is just added to the next chunk and the whole thing gets rescanned. Of course, the delimiter string disappears and would have to be re-inserted if it is needed later on.
It works fine for a bit then the images start breaking up, especially when there is a lot of motion.
Ought to work, but I'm importing only the function node. Can I get the rest to play with?
Also, doesn't this still suffer from @BartButenaers's worry?
Rest is just an exec with ffmpeg -i /path/to/local/1080p.mp4 -f image2pipe pipe:1 set to spawn mode and don't add payload. Need to select your own video file .
And add an inject
Buffer should never be larger than 2 frames/images worth (I think)
