Exec and daemon nodes break up a data stream

General
21

I was much too quick to dismiss this suggestion. It worked brilliantly, once I realized (as I'm sure @dceejay intended) that the "Handle as a stream of messages" option should be enabled.

For the record, and for the benefit of anyone (like myself) who has not seen the split node used this way, I will try to be brief and clear. :slightly_smiling_face: The "Handle as a stream of messages" option changes the way that the split node behaves in two situations. By default (option disabled), strings that do not contain the "Split using" string are sent to the output, and any remaining characters after the last split-using string are also output as a separate message. With the option enabled, the node retains any string that does not end with a split-using string and adds it to the beginning of the next input string. The result is a stream of messages each of which corresponds to a segment of the input between split-using strings, regardless of how the input stream was broken up into separate messages. Neat, and exactly what I needed.

1 Like
22

I'm a lot fuzzy about how to do this. Do I put the split on the output of my exec node and then feed them into a join node to get the full jpeg image?

Mine are binary buffers not strings.

23

Hey @wb666greene,
If you have some spare time, it would be nice if you could share a 'simple' flow that we can use to reproduce the problem.

Was just wondering: suppose you execute a 'curl' command from an exec-node, to load a very large image from the web. If that image also arrives on the output in chunks, then we would at least know that not ffmpeg is responsible for the chunks... Perhaps we could find the cause by elimination this way. Does this makes sense to you?

Had some issues in the past (during development of my node-red-contrib-multipart-stream-decoder node), by the highwatermark level being used by NodeJs streams. However when I look at the Daemon/Exec node in the debugger, it seems that the level is by default on my machine 16K:

image

So I assume this won't cause your issue with image sizes above 65K ...

1 Like
24

I think it (high watermark) is about 16k on windows and 64k on linux.

1 Like
25

How can I change the "highwatermark"? This seems ot explain what I observe Bart's beach scene is ~14K sized images. My HD streams are ~120K typically.

Here is a simple flow to show the problem, but unless you have an rtsp source of 1920x1080 images it won't do anything. My Lorex DVR is not network accessible, so you'd have to change the URL in the top exec node.

The 720p beach scene plays fine, my HD stream is broken up

[{"id":"ae10e3ec.4d2fa","type":"image","z":"e33d1ff4.7f4c9","name":"","width":"1280","x":140,"y":30,"wires":[]},{"id":"301f8afb.348126","type":"exec","z":"e33d1ff4.7f4c9","command":"ffmpeg -f rtsp -i \"rtsp://admin:xyzzy@192.168.2.164:554/cam/realmonitor?channel=4&subtype=0\"  -f image2pipe pipe:1","addpay":false,"append":"","useSpawn":"true","timer":"","oldrc":false,"name":"Decode Lorex RTSP stream","x":640,"y":980,"wires":[["ae10e3ec.4d2fa","710c21de.95036"],[],[]]},{"id":"710c21de.95036","type":"debug","z":"e33d1ff4.7f4c9","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","x":860,"y":965,"wires":[]},{"id":"241ea0c.1cb696","type":"image","z":"e33d1ff4.7f4c9","name":"","width":"1280","x":865,"y":1050,"wires":[]},{"id":"ba23939d.87cbe","type":"exec","z":"e33d1ff4.7f4c9","command":"ffmpeg -f rtsp -i \"rtsp://b1.dnsdojo.com:1935/live/sys3.stream\" -f image2pipe pipe:1","addpay":false,"append":"","useSpawn":"true","timer":"","oldrc":false,"name":"beach","x":600,"y":1065,"wires":[["241ea0c.1cb696"],[],[]]},{"id":"81bd7e74.3cfa7","type":"change","z":"e33d1ff4.7f4c9","name":"","rules":[{"t":"set","p":"kill","pt":"msg","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":340,"y":1150,"wires":[["301f8afb.348126","ba23939d.87cbe"]]},{"id":"c003b3f7.7a843","type":"inject","z":"e33d1ff4.7f4c9","name":"Start stream","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":100,"y":1110,"wires":[["ba23939d.87cbe","301f8afb.348126"]]},{"id":"b32dd9e5.733b38","type":"inject","z":"e33d1ff4.7f4c9","name":"Pause all streams","topic":"","payload":"SIGSTOP","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":120,"y":1150,"wires":[["81bd7e74.3cfa7"]]},{"id":"ba49f593.150278","type":"inject","z":"e33d1ff4.7f4c9","name":"Resume all streams","topic":"","payload":"SIGCONT","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":120,"y":1190,"wires":[["81bd7e74.3cfa7"]]},{"id":"12e87e38.05c532","type":"inject","z":"e33d1ff4.7f4c9","name":"Stop all streams","topic":"","payload":"SIGTERM","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":110,"y":1230,"wires":[["81bd7e74.3cfa7"]]}]
26

If I make an exec node with cpmmand /bin/cat /home/LargePhoto.jpg I get these debug messages:

5/31/2019, 3:13:45 PM[node: 710c21de.95036](http://localhost:1880/#)msg : Object

{ _msgid: "5c6707f0.c7b2f8", topic: "", payload: buffer[65536] }

5/31/2019, 3:13:45 PM[node: 710c21de.95036](http://localhost:1880/#)msg : Object

{ _msgid: "5c6707f0.c7b2f8", topic: "", payload: buffer[65536] }

5/31/2019, 3:13:45 PM[node: 710c21de.95036](http://localhost:1880/#)msg : Object

{ _msgid: "5c6707f0.c7b2f8", topic: "", payload: buffer[65536] }

5/31/2019, 3:13:45 PM[node: 710c21de.95036](http://localhost:1880/#)msg : Object

{ _msgid: "5c6707f0.c7b2f8", topic: "", payload: buffer[23728] }

and the image is garbage.

If I change the /bin/cat image to a small image, it works a single message the size of the small image:

5/31/2019, 3:19:31 PMnode: 710c21de.95036
msg : Object
{ _msgid: "b3e52c0b.f6438", topic: "", payload: buffer[33490] }

So I think we have found the problem, is there a solution? Using smaller images is a non-starter for me.

27

Hey @wb666greene,

you can at least forget about my highwatermark theory :woozy_face:.

When I use curl to get a HD image, it works without problems:

image

[{"id":"90217178.d396","type":"exec","z":"50df184.124f2e8","command":"curl https://images3.alphacoders.com/823/82317.jpg","addpay":true,"append":"","useSpawn":"false","timer":"","oldrc":false,"name":"Get HD image","x":440,"y":980,"wires":[["263274fb.bc1e7c","112ac347.347e7d"],[],[]]},{"id":"7657346e.66b51c","type":"inject","z":"50df184.124f2e8","name":"Start test","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":240,"y":980,"wires":[["90217178.d396"]]},{"id":"263274fb.bc1e7c","type":"debug","z":"50df184.124f2e8","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","x":650,"y":960,"wires":[]},{"id":"112ac347.347e7d","type":"image","z":"50df184.124f2e8","name":"","width":160,"x":660,"y":1000,"wires":[]}]

You can see here that the buffer size is way above the highwatermark of 16K:

image

You can test it yourself, but I don't expect any problems at your side. This convinces me more and more that ffmpeg is responsible for the chuncked output...

28

What happens when you replace replace curl with cat of the large image? Your curl exec node doesn't appear to be in spawn mode i.e. "useSpawn":"false"

29

Not completely sure what you mean. And I'm a bit computerless at the moment. Are you able to reproduce your issue with cat and spawn? If so, please change my flow and share it here.

30

I copied your sample flow and changed the curl exec node to spawn mode and it produced a ton of small fragments:

6/1/2019, 12:25:31 PMnode: 1d2369dd.ca5906
msg.payload : buffer[4096]
[ 255, 216, 255, 224, 0, 16, 74, 70, 73, 70 … ]
6/1/2019, 12:25:32 PMnode: 1d2369dd.ca5906
msg.payload : buffer[12288]
[ 59, 221, 251, 141, 64, 99, 104, 13, 13, 14 … ]
6/1/2019, 12:25:32 PMnode: 1d2369dd.ca5906
msg.payload : buffer[4096]
[ 32, 32, 32, 32, 32, 32, 32, 32, 32, 32 … ]
6/1/2019, 12:25:32 PMnode: 1d2369dd.ca5906
msg.payload : buffer[8192]
[ 189, 212, 231, 146, 226, 194, 199, 252, 71, 211 … ]
6/1/2019, 12:25:32 PMnode: 1d2369dd.ca5906
msg.payload : buffer[4096]
[ 239, 127, 91, 123, 96, 198, 41, 142, 174, 24 … ]
6/1/2019, 12:25:32 PMnode: 1d2369dd.ca5906
msg.payload : buffer[4096]
[ 7, 212, 139, 253, 7, 23, 255, 0, 27, 123 … ]
6/1/2019, 12:25:32 PMnode: 1d2369dd.ca5906
msg.payload : buffer[12288]
[ 247, 174, 189, 215, 29, 67, 250, 143, 246, 254 … ]
6/1/2019, 12:25:32 PMnode: 1d2369dd.ca5906
msg.payload : buffer[4096]
[ 122, 100, 200, 72, 234, 100, 116, 138, 63, 87 … ]
6/1/2019, 12:25:32 PMnode: 1d2369dd.ca5906
msg.payload : buffer[8192]
[ 143, 186, 132, 52, 249, 244, 229, 122, 205, 27 … ]
6/1/2019, 12:25:32 PMnode: 1d2369dd.ca5906
msg.payload : buffer[28672]
[ 167, 167, 95, 255, 208, 213, 191, 54, 177, 211 … ]
6/1/2019, 12:25:32 PMnode: 1d2369dd.ca5906
msg.payload : buffer[16384]
[ 250, 173, 199, 212, 158, 64, 39, 158, 125, 182 … ]
6/1/2019, 12:25:32 PMnode: 1d2369dd.ca5906
msg.payload : buffer[16384]
[ 177, 63, 241, 30, 235, 199, 175, 80, 158, 160 … ]
6/1/2019, 12:25:32 PMnode: 1d2369dd.ca5906
msg.payload : buffer[8192]

Eventually there were some 64K chunks but none larger.

Putting it back to "normal" useSpawn:false behaves like you expect and describe.

6/1/2019, 12:29:51 PMnode: 1d2369dd.ca5906
msg.payload : buffer[1755436]
[ 255, 216, 255, 224, 0, 16, 74, 70, 73, 70 … ]

Only one complete image is sent, when the curl command completes. Since ffmeg "never completes" until we stop it, spawn mode is essential.

I think this 64K highwatermark limit on Linux that dceejay has mentioned explains my results completely. Question then becomes how can we increase this limit?

1 Like
31

I think it’s normally an os level thing (ie tuning the tcp stack). I thought you said the split / join nodes did help re-assemble them though ?

32

That was the OP of this thread who is using json strings that were being split. I don't see how it could work for binary (jpeg) files.

I'm not seeing how the TCP stack settings has anything to do with it, as the same problem happens if I cat a larger than 64K jpeg via an exec node in spawn mode instead of using ffmpeg or curl to source it.

33

It is all coming back in my brain finally. When you have a look at the readme of my multipart encoder, you can see that I have added there in the past an input field to specify the highwatermark level. Will try to have a look at it this week. Currently my portable has disc io issues :worried:

34

You can use the rc output on the exec node (3rd output) to set msg.complete and thus finish the join. Like so

image
image.jpg1588×612 158 KB

Change is like this
image
image.png936×386 9.33 KB

Join is configured like so
image

Meanwhile I have found a slight improvement/fix for the join node - (will be in next 1.0 beta drop) - so you can also just delete the payload rather than have to set it to a buffer of 0;

1 Like
35

This works fine for cat or curl commands which have a defined end point, for ffmpeg decoding the rtsp stream no return code happens until the stream is stopped, but I do get one nice complete image that appears to be a concatenation of all the images in a single message -- the output buffer from about half a minute of running was like 5MB.

I think this approach is doomed, It appears ffmpeg outputs complete jpegs for each frame ~5/sec for my stream but the exec node breaks it up into multiple pieces if the size exceeds 64K.

Now that its dark out, most of my images are ~62K so it mostly works, which has interfered and with and confused me during testing. :frowning:

36

@wb666greene, you may still be able to use the split node with "Handle as a stream of messages" option to assemble complete images. I believe that the jpeg/mpeg format has delimiters that mark the beginning and end of frames, but I don't have a copies of the standards handy. Might be worth a look.

37

Do we have a 100% offline test case flow using ffmpeg we can experiment with?

38

Morning guys,

I confirm that (on my Windows 10 labtop) I can reproduce now the issue, by changing the option to 'spawn' as @wb666greene advised. Then I indeed get 16K chunked output, instead of one complete image:

image

[{"id":"eb2c3372.d484b","type":"exec","z":"22de6542.a4c0fa","command":"curl https://images3.alphacoders.com/823/82317.jpg","addpay":true,"append":"","useSpawn":"true","timer":"","oldrc":false,"name":"Get HD image","x":420,"y":400,"wires":[["8fbcae83.bb55e","e4d0f4a.7ac1108"],[],[]]},{"id":"253573be.dde4ac","type":"inject","z":"22de6542.a4c0fa","name":"Start test","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":220,"y":400,"wires":[["eb2c3372.d484b"]]},{"id":"8fbcae83.bb55e","type":"debug","z":"22de6542.a4c0fa","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","x":630,"y":380,"wires":[]},{"id":"e4d0f4a.7ac1108","type":"image","z":"22de6542.a4c0fa","name":"","width":160,"x":640,"y":420,"wires":[]}]

I had hoped to create a quick fix, but unfortunately ...
When I change (while debugging the 75-exec.js file) the default highWaterMark value of 16K:
image

To 128K:
image
Then I still get a chunked output :woozy_face:. I have also tried it on the _writableState but same result...

But I found on stackoverflow that the highWaterMark values are buffered 'somewhere'. Think we are very close, but we need to find a way to change the highWaterMark somehow (e.g. via options ...).

I'm off now. But if anybody has an idea, don't hesitate to experiment a bit with the above flow ...

39

so using join as above works fine with that example
Note: I changed the command to not append the timestamp (not sure why you had that) - and to not show the progress meter ( -s )

So yes still need an example with ffmpeg and local source of video that I can play with.

[{"id":"bc6c07d1.7f0128","type":"image","z":"64d407f5.647078","name":"","width":160,"x":720,"y":200,"wires":[]},{"id":"cf9559a6.ac1888","type":"join","z":"64d407f5.647078","name":"","mode":"custom","build":"buffer","property":"payload","propertyType":"msg","key":"topic","joiner":"[]","joinerType":"bin","accumulate":false,"timeout":"","count":"","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":550,"y":200,"wires":[["bc6c07d1.7f0128"]]},{"id":"5ec9f2c1.92167c","type":"change","z":"64d407f5.647078","name":"","rules":[{"t":"set","p":"complete","pt":"msg","to":"true","tot":"bool"},{"t":"set","p":"payload","pt":"msg","to":"[]","tot":"bin"}],"action":"","property":"","from":"","to":"","reg":false,"x":360,"y":200,"wires":[["cf9559a6.ac1888"]]},{"id":"7f31e10c.b0253","type":"exec","z":"64d407f5.647078","command":"curl -s https://images3.alphacoders.com/823/82317.jpg","addpay":false,"append":"","useSpawn":"true","timer":"","oldrc":false,"name":"Get HD image","x":300,"y":300,"wires":[["44279b10.f4fa54","cf9559a6.ac1888"],[],["5ec9f2c1.92167c"]]},{"id":"b30a8cd5.77976","type":"inject","z":"64d407f5.647078","name":"Start test","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":120,"y":300,"wires":[["7f31e10c.b0253"]]},{"id":"44279b10.f4fa54","type":"debug","z":"64d407f5.647078","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","x":550,"y":260,"wires":[]}]
40

Hey Dave,
Even if the join works, isn't that a bit inefficient? Because now Node-RED has to handle all the separate messages, which can become quite a lot of extra processing required. Or not ?