Node-red-contrib-ffmpeg-spawn

[kevinGodell]

@BartButenaers

Is it useful to display the installed ffmpeg version number on your config screen, and a warning if ffmpeg not (or not correctly) installed

To get the version, we would have to spawn it ffmpeg -version and split the string and grab the 3rd item from the array as this flow shows:

[{"id":"48e02cfe.7cb014","type":"inject","z":"5967c855.68b978","name":"start","props":[{"p":"action","v":"{\"command\":\"start\"}","vt":"json"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":264,"y":740,"wires":[["ffc6a303.54b73"]]},{"id":"ffc6a303.54b73","type":"ffmpeg-spawn","z":"5967c855.68b978","name":"","outputs":2,"migrate":1e-9,"cmdPath":"","cmdArgs":"[\"-version\"]","cmdOutputs":1,"killSignal":"SIGTERM","x":430,"y":740,"wires":[[],["23ddb71.08b0548"]]},{"id":"abb559f9.d13f48","type":"debug","z":"5967c855.68b978","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":757,"y":740,"wires":[]},{"id":"23ddb71.08b0548","type":"function","z":"5967c855.68b978","name":"","func":"const version = msg.payload.toString().split(' ')[2];\n\nconst message = `version: ${version}`;\n\nnode.status({ fill: 'green', shape: 'dot', text: message });\n\nreturn { payload: version };","outputs":1,"noerr":0,"initialize":"","finalize":"","x":613,"y":740,"wires":[["abb559f9.d13f48"]]}]

I feel that it would be alot of work to try to spawn this as a convenience feature for a little gain. We can just tell confused users to copy/paste this flow and hit the button to output the version. Personally, knowing the version of ffmpeg does not really help me since i don't follow their releases to know what new features might be available.

@BartButenaers

if the checkbox '...' is activated

The args arriving via input should not affect whether there are multiple outputs or a single unified output. I probably didnt explain it right. You will see when I have a chance to build that feature.

@BartButenaers

Can you explain that a bit more

For a moment, I thought about making the topic user settable, but that will be an unecessary feature with little gain. The topics will be status, stdout, stderr, stdio3, stdio4, stdio5 or more if needed.

@krambriw

Yes, also it would be useful if we could define a command that builds and installs ffmpeg with the necessary options, at least for a RPi I think.

i have never tried to build ffmpeg and I applaud those who build and release it for us. Seems very complicated and platform specific. The things i dont know could fill a warehouse. For example, would ffmpeg built on pi3 work for pi4? probably not. Sounds like you would need a team working full time to keep up with it.

[kevinGodell]

That output from ffmpeg is usually what is shown when no args are passed to it, such as an empty array. Are you sure you have the args in the inject node?

[krambriw]

So I did rebuild ffmpeg using the command from above. After this everything works!! So it is depending on how ffmpeg is built. Maybe I did overreach but what would be the simple and correct command to install a ffmpeg version for the RPi that is "good enough". Is "sudo apt-get install ffmpeg" really enough?

[krambriw]

I did a comparison check since I noticed the sample provided did load the cpu heavily. It showed that it is the image2pipe that causes the most load. Just streaming is causing a healthy low level around 6,5% but when image2pipe is active it is close to 50 %

Without pipe2jpeg

With pipe2jpeg

[SuperNinja]

@kevinGodell

Are you sure you have the args in the inject node?

yes , it's a copy-paste of your example "start /restart" inject flow :

[kevinGodell]

@krambriw

but when image2pipe is active it is close to 50 %

That is expected and thank you for reporting it. i was trying to provide an overblown flow example of how much you could output beyond using the exec node. My original command to do this uses much less cpu load because I take advantage of the hardware acceleration in the pi4. I deliberately removed those ffmpeg flags because they would not be compatible for everybody that might try the flow.

If you now have support for hwaccel in ffmpeg, you could try adding -hwaccel rpi -c:v h264_mmal in front of the input -i, although i am not sure if your device supports that particular feature. If it does, then you would also have to modify the -vf output. I just tweaked my setting to get the same ouput, but reduces the cpu load from %65 to %25. This uses the gpu for decoding, but there is no hwaccel for encoding the jpeg, so that is still done on the cpu.

Edit. One thing to mention about hardware acceleration decoding is that there is a limit on the size input of the video W x H. I think most cannot handle an input greater that 1080p and unfortunately ffmpeg will fail without giving a good error to describe the problem. I discovered this by trial an error.

[{"id":"ac4cc611.cee8f8","type":"inject","z":"5967c855.68b978","name":"start args hwaccel","props":[{"p":"action","v":"{\"command\":\"start\",\"args\":[\"-loglevel\",\"error\",\"-nostats\",\"-f\",\"hls\",\"-http_multiple\",\"1\",\"-re\",\"-hwaccel\",\"rpi\",\"-c:v\",\"h264_mmal\",\"-i\",\"https://weather-lh.akamaihd.net/i/twc_1@92006/index_1200_av-p.m3u8?sd=10&rebase=on\",\"-c:v\",\"copy\",\"-c:a\",\"aac\",\"-f\",\"mp4\",\"-movflags\",\"+frag_keyframe+empty_moov+default_base_moof\",\"pipe:1\",\"-progress\",\"pipe:3\",\"-f\",\"image2pipe\",\"-vf\",\"fps=fps=1/2,scale=trunc(iw/4):-2\",\"-vsync\",\"vfr\",\"pipe:4\"]}","vt":"json"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":134,"y":88,"wires":[["93d74c4e.4c8fe"]]},{"id":"42cda3a1.8f558c","type":"inject","z":"5967c855.68b978","name":"start args","props":[{"p":"action","v":"{\"command\":\"start\",\"args\":[\"-loglevel\",\"error\",\"-nostats\",\"-f\",\"hls\",\"-http_multiple\",\"1\",\"-re\",\"-i\",\"https://weather-lh.akamaihd.net/i/twc_1@92006/index_1200_av-p.m3u8?sd=10&rebase=on\",\"-c:v\",\"copy\",\"-c:a\",\"aac\",\"-f\",\"mp4\",\"-movflags\",\"+frag_keyframe+empty_moov+default_base_moof\",\"pipe:1\",\"-progress\",\"pipe:3\",\"-f\",\"image2pipe\",\"-vf\",\"select='eq(pict_type,PICT_TYPE_I)',scale=trunc(iw/4):-2\",\"-vsync\",\"vfr\",\"pipe:4\"]}","vt":"json"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payloadType":"str","x":112,"y":186,"wires":[["93d74c4e.4c8fe"]]}]

Also, how are you getting the cpu load? Can I assume there is some node for that? Can it register the cpu load of a single process using its pid or is that for the entire system?

@SuperNinja

yes , it's a copy-paste of your example "start /restart" inject flow :

I cannot duplicate this error. I guess as the old line goes, "it works for me." But seriously, maybe I had pushed out a newer version and you didnt get the update on your system? Can you try to reinstall it. I made another change last night that automatically puts an empty array in the args settings dialog if you try to use empty string. This is because the typed input is looking for something that can be parsed as json, and if you leave it blank, you will automatically get a red border around the input.

[kevinGodell]

Somewhat related to this node, but more specifically about the usage of ffmpeg, I would someday like to write a list of strategies and example flows. For example, a strategy for creating jpegs from an ip camera would be to use its 2nd output of lower resolution (most ip cams have 2 outputs, high res and low res). The smaller size video will be less of a burden for ffmpeg to decode the h264 and then encode the jpeg. I have seen in the past that most users will have a compulsion to stream the highest quality and fastest fps without understanding that it all comes with a cost. I dont know if this forum allows for such a wiki type page that is locked, except to the original author. Or maybe it will just end up in the readme.

[krambriw]

@kevinGodell

-hwaccel rpi -c:v h264_mmal

Cool!! Thanks! I will try that
For the CPU load monitoring I use the node node-red-contrib-cpu and just presents it in the dashboard. The load is for the entire system, I think top os better to use to monitor single processes

[kevinGodell]

@krambriw

Currently I'm looking at a command like below. But is it enough? Will it rebuild ffmpeg also with GPU support?

I just came across this article referring to building ffmpeg with hardware acceleration support for pi. It doesn't have as many build flags as you have, but it does make reference to pi 2/3, possibly indicating that the feature might be available on older devices. I may try this some day. Compiling ffmpeg with Raspberry Pi hardware acceleration ~ Maniac­land Blog

[krambriw]

Thanks again! I see I had all those flags already in my longer list. When I try your start args hwaccel I get some errors like below. I guess I still miss something. I know the GPU thing works from test I made converting ts streams to mp4

[NULL @ 0x2ed1430] non-existing PPS 0 referenced

[h264 @ 0x2ec85c0] non-existing PPS 0 referenced
[h264 @ 0x2ec85c0] decode_slice_header error
[h264 @ 0x2ec85c0] non-existing PPS 0 referenced
[h264 @ 0x2ec85c0] decode_slice_header error
[h264 @ 0x2ec85c0] no frame!

[krambriw]

Correction, what is logged from the error logs is this:

ffmpeg version git-2021-04-20-718e03e Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 8 (Raspbian 8.3.0-6+rpi1)
  configuration: --extra-cflags=-I/usr/local/include --extra-ldflags=-L/usr/local/lib --extra-libs='-lpthread -lm -latomic' --arch=armel --enable-gmp --enable-gpl --enable-libaom --enable-libass --enable-libdav1d --enable-libdrm --enable-libfdk-aac --enable-libfreetype --enable-libkvazaar --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopus --enable-librtmp --enable-libsnappy --enable-libsoxr --enable-libssh --enable-libvorbis --enable-libvpx --enable-libzimg --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-mmal --enable-nonfree --enable-omx --enable-omx-rpi --enable-version3 --target-os=linux --enable-pthreads --enable-openssl --enable-hardcoded-tables

*`and this`*

  libavutil      56. 73.100 / 56. 73.100
  libavcodec     58.136.101 / 58.136.101
  libavformat    58. 78.100 / 58. 78.100
  libavdevice    58. 14.100 / 58. 14.100
  libavfilter     7.111.100 /  7.111.100
  libswscale      5. 10.100 /  5. 10.100
  libswresample   3. 10.100 /  3. 10.100
  libpostproc    55. 10.100 / 55. 10.100
Hyper fast Audio and Video encoder
usage: ffmpeg [options] [[infile options] -i infile]... {[outfile options] outfile}...

Use -h to get full help or, even better, run 'man ffmpeg

[SuperNinja]

@kevinGodell

Can you try to reinstall it.

All right ! I had done the update well but I had not restarted NodeRed ... now it works by injecting an args

[krambriw]

Some progress made. If I only use -c:v h264_mmal it seems partly working. The pid keeps running but nothing is shown, nothing seems to get piped out

[kevinGodell]

Gpu decoding is even trickier. Not all vf filters work with it, such as that pict type I thing.

[krambriw]

Maybe just academic, anyway a bit interesting. I did some further tests and now I have a visible output from the ui_mp4frag

1. first setup that worked fine was this example. The stream quality is excellent, cpu load just around 8 %. I believe the GPU is involved. If I try the following command sudo /opt/vc/bin/vcdbg reloc stats it shows the GPU has allocated memory:
   Relocatable heap version 4 found at 0x30000000
   total space allocated is 236M, with 236M relocatable, 0 legacy and 0 offline
   0 legacy blocks of size 2359296

{
    "command": "start",
    "args": [
        "-loglevel",
        "error",
        "-nostats",
        "-f",
        "hls",
        "-http_multiple",
        "1",
        "-re",
        "-c:v",
        "h264_mmal",
        "-i",
        "https://weather-lh.akamaihd.net/i/twc_1@92006/index_1200_av-p.m3u8?sd=10&rebase=on",
        "-c:v",
        "copy",
        "-c:a",
        "aac",
        "-f",
        "mp4",
        "-movflags",
        "+frag_keyframe+empty_moov+default_base_moof",
        "pipe:1"
    ]
}

2. In my next test I changed to "-c:v", "h264_omx". This also worked but quality was degraded and the cpu load actually increased, now around 18-20 %. I guess this is because the additional work caused by the h264 decoding (which I guess is completely unnecessary)

{
    "command": "start",
    "args": [
        "-loglevel",
        "error",
        "-nostats",
        "-f",
        "hls",
        "-http_multiple",
        "1",
        "-re",
        "-c:v",
        "h264_mmal",
        "-i",
        "https://weather-lh.akamaihd.net/i/twc_1@92006/index_1200_av-p.m3u8?sd=10&rebase=on",
        "-c:v",
        "h264_omx",
        "-c:a",
        "aac",
        "-f",
        "mp4",
        "-movflags",
        "+frag_keyframe+empty_moov+default_base_moof",
        "pipe:1"
    ]
}

[SuperNinja]

I am testing the RTSP stream read delay difference between the EXE node and the new ffmpeg-spawn node:

ffmpeg-spawn node at the top / EXE node at the bottom

All settings are same :
Conclusion : we see that new ffmpeg-spawn node is delaying the video
@kevinGodell can you do something ?

[kevinGodell]

That is an interesting test. Everybody here always finds new ways to challenge my mind.

At first glance, I would say that it should not be possible for ffmpeg-spawn to output any differently than exec node, but obviously there is some issue here. Internally, I just add a 'data' listener to the pipe, just as exec node does, and simply send the buffer immediately when receiving it.

Assuming, the following to be true:

• same exact parameters passed to ffmpeg (exec and ffmpeg-spawn)
• same exact settings on both ui_mp4frag nodes

Then, output should be the same.

But, I can see a couple possibilities for this without digging into the code:

• If the ip cam source has more than 1 connection, does it struggle to push out video? Some of my cams will actually reboot if I connect a 2nd or 3rd ffmpeg to consume the video.
• Does the order on ui make a difference? For example, when both ui_mp4frag nodes are initialized by the dashboard in some sort of order, one will undoubtedly connect before the other, each fighting to connect via socket io. Perhaps it is a first come - first serve situation. But, you didn't say if you are using the socket or hls video playback.

2 ways that I can think to test:

• Compare 2 exec node to each other (without ffmpeg-spawn)
• Compare 2 ffmpeg-spawn nodes to each other (without exec)

My assumption (hopefully) is that this issue should occur if you run 2 exec nodes to the same cam or 2 ffmpeg-nodes to the same cam.

I might try to test this myself soon. Seems interesting. Perhaps I am handling the buffer data in a bad way.

Edit.

I just ran an experiment and made 4 simultaneous ffmpeg connections to my new amcrest cam(love the cam so far!). 2 used exec, the other 2 ffmpeg-spawn. They were all pretty close to each other, but slightly not in sync (as to be expect). My new cam is awesome and can output segments of 1 second duration, which allows for better relay.

This made me think then, what is your segment duration. Upon looking more closely as the little info beneath the mp4frag node in your gif, i see you have different segment duration output. Must be a different ffmpeg command or perhaps 1 stream is main and the other is sub.

[SuperNinja]

@kevinGodell

But, you didn't say if you are using the socket or hls video playback

@kevinGodell

This made me think then, what is your segment duration. Upon looking more closely as the little info beneath the mp4frag node in your gif, i see you have different segment duration output

the s: and d: are better in EXE . the k: move from False to True always compare to ffmpeg-spanw always True ....

@kevinGodell

Must be a different ffmpeg command or perhaps 1 stream is main and the other is sub.

ffmpeg command are exactly same (copy/paste), so they are all sub stream (see the date OSD on top left: they are same. If main stream, the date is much small)

• Indeed there is a "first come, served" gap but I applied a delay of 250ms to the EXE node so that it is ordered late compared to the others, and despite everything, it is more reactive in the streamming of the video that the ffmpeg-spawn node.

[krambriw]

I tend to agree. I have made a similar test, using the stream from one of my security cameras. What I have noticed are two things that I suspect is influencing

1. The durations are much shorter when they are provided directly from the exec node, varies but many are stable around 0.4 second. From the ffmpeg-spawn, also varies but some stability around 0.8 to 0.9 second

2. It seems the durations for both are increased, actually almost doubles, when both the exec and the ffmpeg-spawn is executing at the same time, same camera stream

In general, shorter durations from the exec node, and this is most likely why you see a delay in comparison of the two. I'm using socket.io but the same result which ever you use

Besides, the starting order did not make any difference in my case. The exec node always produces shorter durations

[kevinGodell]

I really tried to duplicate your results, but cannot. I tried with 2 exec nodes and 2 ffmpeg-spawn nodes all starting a ffmpeg process with the same args and all 4 connecting to the same ip cam (my new amcrest that claims to allow up to 20 connections). Also, my new cam naturally can push out segments with a nice short duration of 1 second. I don't mean to sound like an amcrest salesperson, but so far i am very impressed with it.

All 4 stay perfectly in sync in the backend with the little details beneath mp4frag node showing the segment number (s), duration (d), and whether or not the segment contains a keyframe (k).

As you can see from the exec and ffmpeg-spawn, they have sequential pids. Even though ffmpeg-spawn was added after the exec nodes, it always seems to spawn faster and gets a lower pid. Although, I am not knowledgeable of the order that nodered initializes nodes.

If you are getting drastically different segment durations that line up very close to 2 seconds durations and 0.5 seconds duration, that really sounds like a -movflags +frag_keyframe versus +frag_every_frame situation. To be 100% sure, please investigate the movflags in your args. And even if you are so kind, exports the nodes in question while redacting any private info it may include.

Looking closer again at the shared gif, I can see that the keyframe value toggles from true or false, while the one with 2 seconds duration is always true. That really confirms for me that there are 2 sets of ffmpeg args being used. Maybe I had pushed an update at some point and the internal settings of the node got messed up and the args are not what they appear to be. But since I wrote the code, i know that I only push out the segments once they are complete and also tag them with a keyframe value.