# Exec and daemon nodes break up a data stream

**URL:** <https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695>\
**Category:** General\
**Created:** [30 April 2019 21:10 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695 "2019-04-30T21:10:33Z")\
**Posts on this page:** 20\
**Page:** 3

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [2 June 2019 09:43 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/41 "2019-06-02T09:43:22Z")

</div>

> [@BartButenaers](#):
>
> isn't that a bit inefficient?

Trivial I think compared to the amount of work ffmpeg is doing decoding the rtsp stream.

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [2 June 2019 11:41 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/42 "2019-06-02T11:41:29Z")

</div>

> [@dceejay](#):
>
> The exec node just outputs "chunks" of data as given to it by the operating system. We don't have any control over it

Of course Dave is always right, but like my wife always says: "_Trust is good, but control is better..._". So to gather some evidence, I have quickly debugged where the chunks (in the "data" event of the Exec-node) are coming from. Seems the chunks have indeed already a size of 65K when the come out of the **socket** :

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/2X/e/e993fc2b3276273e9f0f95e5f22459b29d515cc3.png)

So my highWaterMark level of 128K is never reached, and has no effect at all...  
And for the non-believers among us, here is an extra [explanation](https://github.com/nodejs/help/issues/1265) from the NodeJs team, that the chunk size is determined by the operating system.

![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/2X/5/540d16e947828b46269ffd2ef6f4325180be1e56.png) **...**

---

<div class="post-metadata">

**Author:** ![wb666greene](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/wb666greene/32/6534_2.png) [@wb666greene](https://discourse.nodered.org/u/wb666greene)\
**Post date:** [2 June 2019 17:02 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/43 "2019-06-02T17:02:26Z")

</div>

> [@dceejay](#):
>
> Do we have a 100% offline test case flow using ffmpeg we can experiment with?

replace the ffmpeg (or curl command in Bart's example) with:

ffmpeg -i /path/to/local/1080p.mp4 -f image2pipe pipe:1

The size of the output jpegs will depend on the bit rate of the videofile, but the imageOutput display will break up, flash, or otherwise misbehave when the jpeg frame exceeds the highwatermark.

If Bart's RIP is correct, too bad, as this would have been an extremely slick way to convert rtsp streams into MQTT buffers for frame by frame processing.

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [2 June 2019 20:30 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/44 "2019-06-02T20:30:47Z")

</div>

> [@wb666greene](#):
>
> If Bart's RIP is correct, too bad, as this would have been an extremely slick way to convert rtsp streams

Yes but only the idea (to adapt the Exec-node to generate a single chunk per image) is RIP. But if we can use the Join or Split nodes - as Dave and Michael suggest - _efficiently_ to concatenate the chunks into a single image, then we are back on track I assume. I have never used those nodes, so the others will be of more help to you from here on ...

---

<div class="post-metadata">

**Author:** ![wb666greene](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/wb666greene/32/6534_2.png) [@wb666greene](https://discourse.nodered.org/u/wb666greene)\
**Post date:** [2 June 2019 20:49 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/45 "2019-06-02T20:49:00Z")

</div>

I think there is a fundamental issue with trying to use join and split nodes on binary buffers as produced by ffmpeg piping frames to stdout.

There is a set of "magic bytes" at the start of standardized compressed file formats, jpeg for example generally start with 0xFF 0xD8 0xFF but there is nothing to signal the end of the file and as far as I know nothing in the jpeg compression standard that would prevent this string from occurring somewhere in the compressed data stream.

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [2 June 2019 21:07 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/46 "2019-06-02T21:07:10Z")

</div>

If the stream is a sequence of images there must be some way for the receiver to know when one image finishes and the next starts.

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [2 June 2019 21:07 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/47 "2019-06-02T21:07:33Z")

</div>

But can't we just do it like this: we collect data until we see the magic byte sequence. Then we now that the previous image is complete, so we can send an output image. And we start collecting the bytes of the next image. Think it is enough to have the start of the image, since that is also the end of the previous image ... I do something similar in my node-red-contrib-mutlipart-stream decoder, to find the boundary (i.e. separator) between jpegs in an mjpeg stream. Only disadvantage is that it will consume cpu to find the magic bytes in the buffer !!!

---

<div class="post-metadata">

**Author:** ![drmibell](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/drmibell/32/8424_2.png) [@drmibell](https://discourse.nodered.org/u/drmibell)\
**Post date:** [2 June 2019 21:16 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/48 "2019-06-02T21:16:17Z")

</div>

For what it's worth the [Wikipedia](https://en.wikipedia.org/wiki/JPEG#Syntax_and_structure) article on jpeg says that

- The start of an image is marked by 0xFF 0xD8

- The end of an image is marked by 0xFF 0xD9

- When 0xFF occurs in the data stream, the encoder inserts 0x00 after it so that it won't be mistaken for a marker

It might be worth trying to `split` on 0xFF 0xD9 with "Handle as a stream of messages" enabled. The 0xFF 0xD9 bytes would have to be added back to the message, but it should contain an entire image.

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [2 June 2019 21:21 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/49 "2019-06-02T21:21:01Z")

</div>

> [@drmibell](#):
>
> bytes would have to be added back to the message

Hey Michael,  
We will have to do things like that with caution. If we do it incorrectly, NodeJs will create a entirely new buffer that is long enough to store the original buffer + the new magic byte sequence. But that means NodeJs will copy the entire image bytes behind our backs, which is not what we want (for performance).

P.S. In my multipart stream decoder I had in the beginning lots of issues, when the magic bytes are divided across the boundary between two chunks. Part of the magic bytes are in at the end of chunk N and the rest of the magic bytes are at the start of the chunk N+1. Found a solution at last, but just something we have to think about...

---

<div class="post-metadata">

**Author:** ![drmibell](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/drmibell/32/8424_2.png) [@drmibell](https://discourse.nodered.org/u/drmibell)\
**Post date:** [2 June 2019 21:27 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/50 "2019-06-02T21:27:22Z")

</div>

> [@BartButenaers](#):
>
> NodeJs will create a entirely new buffer

I was just about to post about that. I suspect that Node-RED might have already duplicated buffers created by ffmpeg, so performance is certain to be an issue.

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [2 June 2019 21:33 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/51 "2019-06-02T21:33:44Z")

</div>

> [@BartButenaers](#):
>
> when the magic bytes are divided across the boundary between two chunks.

Don't know it you can find byte sequences (even across chunk boundaries) with the Split or Join nodes? Anyway I developed last month an alpha version of a Buffer-node, which contains a.o. this kind of functionality. If anybody is interested I can publish next week a test version of it ...

---

<div class="post-metadata">

**Author:** ![wb666greene](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/wb666greene/32/6534_2.png) [@wb666greene](https://discourse.nodered.org/u/wb666greene)\
**Post date:** [2 June 2019 21:40 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/52 "2019-06-02T21:40:00Z")

</div>

I don't know. poking around on some jpeg images with GHEX "dump" program there are quite a few 0xFF characters incluiding some followed by 0xD8 and others followed by 0xD9 I'd be expecting false start and stop flags.

However looking at a few jpeg images produced by:  
`ffmpeg -f rtsp -i "rtsp://admin:xyzzy@192.168.2.164:554/cam/realmonitor?channel=4&subtype=0" -f image2 out_img-%03d.jpe`

from my security DVR rtsp streams, it does look like 0xFFD8 always starts the file and 0xFFD9 always ends it. So perhaps it could work in this situation,

---

<div class="post-metadata">

**Author:** ![drmibell](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/drmibell/32/8424_2.png) [@drmibell](https://discourse.nodered.org/u/drmibell)\
**Post date:** [2 June 2019 21:40 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/53 "2019-06-02T21:40:13Z")

</div>

> [@BartButenaers](#):
>
> even across chunk boundaries

Good point, and I missed it. The `split` node will accept multi-character (and I assume multi-byte) delimiters, and the "Handle as a stream of messages" logic _ought_ to put things right, but I will write a test flow to look at this.

---

<div class="post-metadata">

**Author:** ![wb666greene](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/wb666greene/32/6534_2.png) [@wb666greene](https://discourse.nodered.org/u/wb666greene)\
**Post date:** [2 June 2019 21:45 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/54 "2019-06-02T21:45:54Z")

</div>

If you figure it out I will certainly test it in short order!

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [2 June 2019 22:08 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/55 "2019-06-02T22:08:51Z")

</div>

something like this maybe

```auto
[{"id":"513524c9.1e097c","type":"function","z":"64d407f5.647078","name":"","func":"var b = Buffer.concat([(context.buf || Buffer.from(\"\")), msg.payload]);\n\nvar i = b.indexOf(\"FFD9\", 0, \"hex\");\nif (i >= 0) {\n var frame = b.slice(0,i+1);\n context.buf = b.slice(i+2);\n node.send({payload:frame});\n}\nelse {\n context.buf = b;\n}\nreturn null;","outputs":1,"noerr":0,"x":600,"y":500,"wires":[["15aa84d7.ed77ab"]]}]

```

 ![](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/2X/2/244b5f2b10ffda0b5e15e8b188f03b3131aa543d.gif)

---

<div class="post-metadata">

**Author:** ![drmibell](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/drmibell/32/8424_2.png) [@drmibell](https://discourse.nodered.org/u/drmibell)\
**Post date:** [2 June 2019 22:10 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/56 "2019-06-02T22:10:25Z")

</div>

> [@drmibell](#):
>
> the "Handle as a stream of messages" logic _ought_ to put things right

Yes, I tested with multi-character delimiters, and it works ok. I think a chunk containing a partial delimiter is just added to the next chunk and the whole thing gets rescanned. Of course, the delimiter string disappears and would have to be re-inserted if it is needed later on.

---

<div class="post-metadata">

**Author:** ![wb666greene](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/wb666greene/32/6534_2.png) [@wb666greene](https://discourse.nodered.org/u/wb666greene)\
**Post date:** [2 June 2019 22:29 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/57 "2019-06-02T22:29:50Z")

</div>

It works fine for a bit then the images start breaking up, especially when there is a lot of motion.

---

<div class="post-metadata">

**Author:** ![drmibell](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/drmibell/32/8424_2.png) [@drmibell](https://discourse.nodered.org/u/drmibell)\
**Post date:** [2 June 2019 22:30 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/58 "2019-06-02T22:30:04Z")

</div>

> [@dceejay](#):
>
> something like this maybe

Ought to work, but I'm importing only the `function` node. Can I get the rest to play with?

Also, doesn't this still suffer from @BartButenaers's worry?

> [@BartButenaers](#):
>
> NodeJs will create a entirely new buffer that is long enough to store the original buffer + the new magic byte sequence.

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [2 June 2019 22:45 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/59 "2019-06-02T22:45:18Z")

</div>

Rest is just an exec with `ffmpeg -i /path/to/local/1080p.mp4 -f image2pipe pipe:1` set to spawn mode and don't add payload. Need to select your own video file .  
And add an inject

Buffer should never be larger than 2 frames/images worth (I think)

---

<div class="post-metadata">

**Author:** ![drmibell](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/drmibell/32/8424_2.png) [@drmibell](https://discourse.nodered.org/u/drmibell)\
**Post date:** [2 June 2019 22:47 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/60 "2019-06-02T22:47:35Z")

</div>

> [@dceejay](#):
>
> Buffer should never be larger than 2 frames/images worth (I think)

Agreed, but since @BartButenaers asked...

[Previous page](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695.md?page=2)

[Next page](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695.md?page=4)