# Exec and daemon nodes break up a data stream
**URL:** https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695
**Category:** General
**Created:** [30 April 2019 21:10 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695 "2019-04-30T21:10:33Z")
**Posts on this page:** 20
**Page:** 2
### Author:  [@drmibell](https://discourse.nodered.org/u/drmibell)
#### Post date: [28 May 2019 20:35 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/21 "2019-05-28T20:35:07Z")
> [@dceejay](#):
>
> Would the split node do this for you?
I was much too quick to dismiss this suggestion. It worked brilliantly, once I realized (as I'm sure @dceejay intended) that the "Handle as a stream of messages" option should be enabled.
For the record, and for the benefit of anyone (like myself) who has not seen the `split` node used this way, I will try to be brief and clear. π The "Handle as a stream of messages" option changes the way that the `split` node behaves in two situations. By default (option disabled), strings that do not contain the "Split using" string are sent to the output, and any remaining characters after the last split-using string are also output as a separate message. With the option enabled, the node retains any string that does not end with a split-using string and adds it to the beginning of the next input string. The result is a stream of messages each of which corresponds to a segment of the input between split-using strings, regardless of how the input stream was broken up into separate messages. Neat, and exactly what I needed.
---
### Author:  [@wb666greene](https://discourse.nodered.org/u/wb666greene)
#### Post date: [30 May 2019 22:05 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/22 "2019-05-30T22:05:52Z")
I'm a lot fuzzy about how to do this. Do I put the split on the output of my exec node and then feed them into a join node to get the full jpeg image?
Mine are binary buffers not strings.
---
### Author:  [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)
#### Post date: [30 May 2019 23:00 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/23 "2019-05-30T23:00:03Z")
Hey @wb666greene,
If you have some spare time, it would be nice if you could share a 'simple' flow that we can use to reproduce the problem.
Was just wondering: suppose you execute a '_ **curl** _' command from an exec-node, to load a very large image from the web. If that image also arrives on the output in chunks, then we would at least know that not ffmpeg is responsible for the chunks... Perhaps we could find the cause by elimination this way. Does this makes sense to you?
Had some issues in the past (during development of my node-red-contrib-multipart-stream-decoder node), by the _ **highwatermark** _ level being used by NodeJs streams. However when I look at the Daemon/Exec node in the debugger, it seems that the level is by default on my machine 16K:
So I assume this won't cause your issue with image sizes above 65K ...
---
### Author:  [@dceejay](https://discourse.nodered.org/u/dceejay)
#### Post date: [30 May 2019 23:14 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/24 "2019-05-30T23:14:38Z")
I think it (high watermark) is about 16k on windows and 64k on linux.
---
### Author:  [@wb666greene](https://discourse.nodered.org/u/wb666greene)
#### Post date: [31 May 2019 19:48 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/25 "2019-05-31T19:48:55Z")
How can I change the "highwatermark"? This seems ot explain what I observe Bart's beach scene is ~14K sized images. My HD streams are ~120K typically.
Here is a simple flow to show the problem, but unless you have an rtsp source of 1920x1080 images it won't do anything. My Lorex DVR is not network accessible, so you'd have to change the URL in the top exec node.
The 720p beach scene plays fine, my HD stream is broken up
```auto
[{"id":"ae10e3ec.4d2fa","type":"image","z":"e33d1ff4.7f4c9","name":"","width":"1280","x":140,"y":30,"wires":[]},{"id":"301f8afb.348126","type":"exec","z":"e33d1ff4.7f4c9","command":"ffmpeg -f rtsp -i \"rtsp://admin:xyzzy@192.168.2.164:554/cam/realmonitor?channel=4&subtype=0\" -f image2pipe pipe:1","addpay":false,"append":"","useSpawn":"true","timer":"","oldrc":false,"name":"Decode Lorex RTSP stream","x":640,"y":980,"wires":[["ae10e3ec.4d2fa","710c21de.95036"],[],[]]},{"id":"710c21de.95036","type":"debug","z":"e33d1ff4.7f4c9","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","x":860,"y":965,"wires":[]},{"id":"241ea0c.1cb696","type":"image","z":"e33d1ff4.7f4c9","name":"","width":"1280","x":865,"y":1050,"wires":[]},{"id":"ba23939d.87cbe","type":"exec","z":"e33d1ff4.7f4c9","command":"ffmpeg -f rtsp -i \"rtsp://b1.dnsdojo.com:1935/live/sys3.stream\" -f image2pipe pipe:1","addpay":false,"append":"","useSpawn":"true","timer":"","oldrc":false,"name":"beach","x":600,"y":1065,"wires":[["241ea0c.1cb696"],[],[]]},{"id":"81bd7e74.3cfa7","type":"change","z":"e33d1ff4.7f4c9","name":"","rules":[{"t":"set","p":"kill","pt":"msg","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":340,"y":1150,"wires":[["301f8afb.348126","ba23939d.87cbe"]]},{"id":"c003b3f7.7a843","type":"inject","z":"e33d1ff4.7f4c9","name":"Start stream","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":100,"y":1110,"wires":[["ba23939d.87cbe","301f8afb.348126"]]},{"id":"b32dd9e5.733b38","type":"inject","z":"e33d1ff4.7f4c9","name":"Pause all streams","topic":"","payload":"SIGSTOP","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":120,"y":1150,"wires":[["81bd7e74.3cfa7"]]},{"id":"ba49f593.150278","type":"inject","z":"e33d1ff4.7f4c9","name":"Resume all streams","topic":"","payload":"SIGCONT","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":120,"y":1190,"wires":[["81bd7e74.3cfa7"]]},{"id":"12e87e38.05c532","type":"inject","z":"e33d1ff4.7f4c9","name":"Stop all streams","topic":"","payload":"SIGTERM","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":110,"y":1230,"wires":[["81bd7e74.3cfa7"]]}]
```
---
### Author:  [@wb666greene](https://discourse.nodered.org/u/wb666greene)
#### Post date: [31 May 2019 20:20 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/26 "2019-05-31T20:20:48Z")
If I make an exec node with cpmmand /bin/cat /home/LargePhoto.jpg I get these debug messages:
```auto
5/31/2019, 3:13:45 PM[node: 710c21de.95036](http://localhost:1880/#)msg : Object
{ _msgid: "5c6707f0.c7b2f8", topic: "", payload: buffer[65536] }
5/31/2019, 3:13:45 PM[node: 710c21de.95036](http://localhost:1880/#)msg : Object
{ _msgid: "5c6707f0.c7b2f8", topic: "", payload: buffer[65536] }
5/31/2019, 3:13:45 PM[node: 710c21de.95036](http://localhost:1880/#)msg : Object
{ _msgid: "5c6707f0.c7b2f8", topic: "", payload: buffer[65536] }
5/31/2019, 3:13:45 PM[node: 710c21de.95036](http://localhost:1880/#)msg : Object
{ _msgid: "5c6707f0.c7b2f8", topic: "", payload: buffer[23728] }
```
and the image is garbage.
If I change the /bin/cat image to a small image, it works a single message the size of the small image:
```auto
5/31/2019, 3:19:31 PMnode: 710c21de.95036
msg : Object
{ _msgid: "b3e52c0b.f6438", topic: "", payload: buffer[33490] }
```
So I think we have found the problem, is there a solution? Using smaller images is a non-starter for me.
---
### Author:  [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)
#### Post date: [31 May 2019 20:39 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/27 "2019-05-31T20:39:06Z")
Hey @wb666greene,
you can at least forget about my highwatermark theory π₯΄.
When I use **curl** to get a HD image, it works without problems:
```auto
[{"id":"90217178.d396","type":"exec","z":"50df184.124f2e8","command":"curl https://images3.alphacoders.com/823/82317.jpg","addpay":true,"append":"","useSpawn":"false","timer":"","oldrc":false,"name":"Get HD image","x":440,"y":980,"wires":[["263274fb.bc1e7c","112ac347.347e7d"],[],[]]},{"id":"7657346e.66b51c","type":"inject","z":"50df184.124f2e8","name":"Start test","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":240,"y":980,"wires":[["90217178.d396"]]},{"id":"263274fb.bc1e7c","type":"debug","z":"50df184.124f2e8","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","x":650,"y":960,"wires":[]},{"id":"112ac347.347e7d","type":"image","z":"50df184.124f2e8","name":"","width":160,"x":660,"y":1000,"wires":[]}]
```
You can see here that the buffer size is way above the highwatermark of 16K:
You can test it yourself, but I don't expect any problems at your side. This convinces me more and more that ffmpeg is responsible for the chuncked output...
---
### Author:  [@wb666greene](https://discourse.nodered.org/u/wb666greene)
#### Post date: [1 June 2019 13:12 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/28 "2019-06-01T13:12:20Z")
> [@BartButenaers](#):
>
> you can at least forget about my highwatermark theory
What happens when you replace replace curl with cat of the large image? Your curl exec node doesn't appear to be in spawn mode i.e. "useSpawn":"false"
---
### Author:  [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)
#### Post date: [1 June 2019 15:52 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/29 "2019-06-01T15:52:26Z")
> [@wb666greene](#):
>
> replace replace curl with cat
Not completely sure what you mean. And I'm a bit computerless at the moment. Are you able to reproduce your issue with _cat_ and _spawn_? If so, please change my flow and share it here.
---
### Author:  [@wb666greene](https://discourse.nodered.org/u/wb666greene)
#### Post date: [1 June 2019 17:34 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/30 "2019-06-01T17:34:56Z")
> [@BartButenaers](#):
>
> If so, please change my flow and share it here.
I copied your sample flow and changed the curl exec node to spawn mode and it produced a ton of small fragments:
```auto
6/1/2019, 12:25:31 PMnode: 1d2369dd.ca5906
msg.payload : buffer[4096]
[255, 216, 255, 224, 0, 16, 74, 70, 73, 70 β¦]
6/1/2019, 12:25:32 PMnode: 1d2369dd.ca5906
msg.payload : buffer[12288]
[59, 221, 251, 141, 64, 99, 104, 13, 13, 14 β¦]
6/1/2019, 12:25:32 PMnode: 1d2369dd.ca5906
msg.payload : buffer[4096]
[32, 32, 32, 32, 32, 32, 32, 32, 32, 32 β¦]
6/1/2019, 12:25:32 PMnode: 1d2369dd.ca5906
msg.payload : buffer[8192]
[189, 212, 231, 146, 226, 194, 199, 252, 71, 211 β¦]
6/1/2019, 12:25:32 PMnode: 1d2369dd.ca5906
msg.payload : buffer[4096]
[239, 127, 91, 123, 96, 198, 41, 142, 174, 24 β¦]
6/1/2019, 12:25:32 PMnode: 1d2369dd.ca5906
msg.payload : buffer[4096]
[7, 212, 139, 253, 7, 23, 255, 0, 27, 123 β¦]
6/1/2019, 12:25:32 PMnode: 1d2369dd.ca5906
msg.payload : buffer[12288]
[247, 174, 189, 215, 29, 67, 250, 143, 246, 254 β¦]
6/1/2019, 12:25:32 PMnode: 1d2369dd.ca5906
msg.payload : buffer[4096]
[122, 100, 200, 72, 234, 100, 116, 138, 63, 87 β¦]
6/1/2019, 12:25:32 PMnode: 1d2369dd.ca5906
msg.payload : buffer[8192]
[143, 186, 132, 52, 249, 244, 229, 122, 205, 27 β¦]
6/1/2019, 12:25:32 PMnode: 1d2369dd.ca5906
msg.payload : buffer[28672]
[167, 167, 95, 255, 208, 213, 191, 54, 177, 211 β¦]
6/1/2019, 12:25:32 PMnode: 1d2369dd.ca5906
msg.payload : buffer[16384]
[250, 173, 199, 212, 158, 64, 39, 158, 125, 182 β¦]
6/1/2019, 12:25:32 PMnode: 1d2369dd.ca5906
msg.payload : buffer[16384]
[177, 63, 241, 30, 235, 199, 175, 80, 158, 160 β¦]
6/1/2019, 12:25:32 PMnode: 1d2369dd.ca5906
msg.payload : buffer[8192]
```
Eventually there were some 64K chunks but none larger.
Putting it back to "normal" useSpawn:false behaves like you expect and describe.
```auto
6/1/2019, 12:29:51 PMnode: 1d2369dd.ca5906
msg.payload : buffer[1755436]
[255, 216, 255, 224, 0, 16, 74, 70, 73, 70 β¦]
```
Only one complete image is sent, when the curl command completes. Since ffmeg "never completes" until we stop it, spawn mode is essential.
I think this 64K highwatermark limit on Linux that dceejay has mentioned explains my results completely. Question then becomes how can we increase this limit?
---
### Author:  [@dceejay](https://discourse.nodered.org/u/dceejay)
#### Post date: [1 June 2019 18:41 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/31 "2019-06-01T18:41:50Z")
I think itβs normally an os level thing (ie tuning the tcp stack). I thought you said the split / join nodes did help re-assemble them though ?
---
### Author:  [@wb666greene](https://discourse.nodered.org/u/wb666greene)
#### Post date: [1 June 2019 21:38 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/32 "2019-06-01T21:38:01Z")
That was the OP of this thread who is using json strings that were being split. I don't see how it could work for binary (jpeg) files.
I'm not seeing how the TCP stack settings has anything to do with it, as the same problem happens if I cat a larger than 64K jpeg via an exec node in spawn mode instead of using ffmpeg or curl to source it.
---
### Author:  [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)
#### Post date: [1 June 2019 21:50 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/33 "2019-06-01T21:50:44Z")
> [@wb666greene](#):
>
> Question then becomes how can we increase this limit?
It is all coming back in my brain finally. When you have a look at the [readme](https://github.com/bartbutenaers/node-red-contrib-multipart-stream-encoder/blob/master/README.md#adaptive-streaming) of my multipart encoder, you can see that I have added there in the past an input field to specify the highwatermark level. Will try to have a look at it this week. Currently my portable has disc io issues π
---
### Author:  [@dceejay](https://discourse.nodered.org/u/dceejay)
#### Post date: [1 June 2019 22:58 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/34 "2019-06-01T22:58:50Z")
You can use the rc output on the exec node (3rd output) to set msg.complete and thus finish the join. Like so
Change is like this
Join is configured like so
Meanwhile I have found a slight improvement/fix for the join node - (will be in next 1.0 beta drop) - so you can also just delete the payload rather than have to set it to a buffer of 0;
---
### Author:  [@wb666greene](https://discourse.nodered.org/u/wb666greene)
#### Post date: [2 June 2019 01:59 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/35 "2019-06-02T01:59:52Z")
This works fine for cat or curl commands which have a defined end point, for ffmpeg decoding the rtsp stream no return code happens until the stream is stopped, but I do get one nice complete image that appears to be a concatenation of all the images in a single message -- the output buffer from about half a minute of running was like 5MB.
I think this approach is doomed, It appears ffmpeg outputs complete jpegs for each frame ~5/sec for my stream but the exec node breaks it up into multiple pieces if the size exceeds 64K.
Now that its dark out, most of my images are ~62K so it mostly works, which has interfered and with and confused me during testing. βΉ
---
### Author:  [@drmibell](https://discourse.nodered.org/u/drmibell)
#### Post date: [2 June 2019 02:33 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/36 "2019-06-02T02:33:25Z")
@wb666greene, you may still be able to use the `split` node with "Handle as a stream of messages" option to assemble complete images. I believe that the jpeg/mpeg format has delimiters that mark the beginning and end of frames, but I don't have a copies of the standards handy. Might be worth a look.
---
### Author:  [@dceejay](https://discourse.nodered.org/u/dceejay)
#### Post date: [2 June 2019 06:38 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/37 "2019-06-02T06:38:41Z")
Do we have a 100% offline test case flow using ffmpeg we can experiment with?
---
### Author:  [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)
#### Post date: [2 June 2019 07:20 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/38 "2019-06-02T07:20:36Z")
Morning guys,
I confirm that (on my Windows 10 labtop) I can reproduce now the issue, by changing the option to '_spawn_' as @wb666greene advised. Then I indeed get 16K chunked output, instead of one complete image:
```auto
[{"id":"eb2c3372.d484b","type":"exec","z":"22de6542.a4c0fa","command":"curl https://images3.alphacoders.com/823/82317.jpg","addpay":true,"append":"","useSpawn":"true","timer":"","oldrc":false,"name":"Get HD image","x":420,"y":400,"wires":[["8fbcae83.bb55e","e4d0f4a.7ac1108"],[],[]]},{"id":"253573be.dde4ac","type":"inject","z":"22de6542.a4c0fa","name":"Start test","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":220,"y":400,"wires":[["eb2c3372.d484b"]]},{"id":"8fbcae83.bb55e","type":"debug","z":"22de6542.a4c0fa","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","x":630,"y":380,"wires":[]},{"id":"e4d0f4a.7ac1108","type":"image","z":"22de6542.a4c0fa","name":"","width":160,"x":640,"y":420,"wires":[]}]
```
I had hoped to create a quick fix, but unfortunately ...
When I change (while debugging the 75-exec.js file) the default highWaterMark value of 16K:
To 128K:
Then _I still get a chunked output_ π₯΄. I have also tried it on the \_writableState but same result...
But I found on [stackoverflow](https://stackoverflow.com/questions/22569114/node-js-child-process-spawn-stdout-lower-highwatermark) that the highWaterMark values are buffered 'somewhere'. Think we are very close, but we need to find a way to change the highWaterMark somehow (e.g. via options ...).
I'm off now. But if anybody has an idea, don't hesitate to experiment a bit with the above flow ...
---
### Author:  [@dceejay](https://discourse.nodered.org/u/dceejay)
#### Post date: [2 June 2019 08:57 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/39 "2019-06-02T08:57:59Z")
so using join as above works fine with that example
Note: I changed the command to not append the timestamp (not sure why you had that) - and to not show the progress meter ( -s )
So yes still need an example with ffmpeg and local source of video that I can play with.
```auto
[{"id":"bc6c07d1.7f0128","type":"image","z":"64d407f5.647078","name":"","width":160,"x":720,"y":200,"wires":[]},{"id":"cf9559a6.ac1888","type":"join","z":"64d407f5.647078","name":"","mode":"custom","build":"buffer","property":"payload","propertyType":"msg","key":"topic","joiner":"[]","joinerType":"bin","accumulate":false,"timeout":"","count":"","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":550,"y":200,"wires":[["bc6c07d1.7f0128"]]},{"id":"5ec9f2c1.92167c","type":"change","z":"64d407f5.647078","name":"","rules":[{"t":"set","p":"complete","pt":"msg","to":"true","tot":"bool"},{"t":"set","p":"payload","pt":"msg","to":"[]","tot":"bin"}],"action":"","property":"","from":"","to":"","reg":false,"x":360,"y":200,"wires":[["cf9559a6.ac1888"]]},{"id":"7f31e10c.b0253","type":"exec","z":"64d407f5.647078","command":"curl -s https://images3.alphacoders.com/823/82317.jpg","addpay":false,"append":"","useSpawn":"true","timer":"","oldrc":false,"name":"Get HD image","x":300,"y":300,"wires":[["44279b10.f4fa54","cf9559a6.ac1888"],[],["5ec9f2c1.92167c"]]},{"id":"b30a8cd5.77976","type":"inject","z":"64d407f5.647078","name":"Start test","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":120,"y":300,"wires":[["7f31e10c.b0253"]]},{"id":"44279b10.f4fa54","type":"debug","z":"64d407f5.647078","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","x":550,"y":260,"wires":[]}]
```
---
### Author:  [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)
#### Post date: [2 June 2019 09:23 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/40 "2019-06-02T09:23:30Z")
> [@dceejay](#):
>
> so using join as above works fine with that example
Hey Dave,
Even if the join works, isn't that a bit inefficient? Because now Node-RED has to handle all the separate messages, which can become quite a lot of extra processing required. Or not ?
[Previous page](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695.md?page=1)
[Next page](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695.md?page=3)