# Is there any way to get frames out of the ffmpeg node?

**URL:** <https://discourse.nodered.org/t/is-there-any-way-to-get-frames-out-of-the-ffmpeg-node/77576>\
**Category:** General\
**Created:** [15 April 2023 00:05 UTC](https://discourse.nodered.org/t/is-there-any-way-to-get-frames-out-of-the-ffmpeg-node/77576 "2023-04-15T00:05:35Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![Seba3Fi](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/seba3fi/32/78891_2.png) [@Seba3Fi](https://discourse.nodered.org/u/Seba3Fi)\
**Post date:** [15 April 2023 00:05 UTC](https://discourse.nodered.org/t/is-there-any-way-to-get-frames-out-of-the-ffmpeg-node/77576/1 "2023-04-15T00:05:35Z")

</div>

Dear @kevinGodell, thanks for posting the ffmpeg module, I am testing it in the context of a home made security system (learning process)

My current workflow is something like this:  
-In Scrypted the rstp flows are processed, and motion is detected. This triggers MQTT events that I receive on Node Red. Scrypted also provides some features like exposing to Apple HomeKit and others, and rebroadcasting camera traffic via rtsp.  
-The second layer is Node Red. Today I am getting a picture from every camera via http, and procesing it with Tensorflow to detect objects of interst.

Instead of fetching images via http (that provides a lot of timeouts) I would rather use the rtsp stream to get 1 picture at a time via your ffmpeg node.

I have used ffmpeg from the command line, and it provides the frames, which I could save to disk or pipe into another ffmpeg command for further processing. But that leaves the image let's say at the operating system level, and I would rather have the image processed on node red.

Could you please expand the examples so that on e of the outputs shows 1 frame?  
I am currently running the code on a MAC Mini (intel, 2014)

[![WhatsApp Image 2023-04-14 at 11 21 43](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/5/a/5a9fb945f3c6827b8cf1c2da6b215a81ab55aebe.jpeg)](https://user-images.githubusercontent.com/15160805/232071373-a2743f4b-cebb-4a44-b8fa-f309d09ab115.jpeg)

Thanks in advance,  
Sebastián|

---

<div class="post-metadata">

**Author:** ![kevinGodell](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/kevingodell/32/27040_2.png) [@kevinGodell](https://discourse.nodered.org/u/kevinGodell)\
**Post date:** [15 April 2023 01:09 UTC](https://discourse.nodered.org/t/is-there-any-way-to-get-frames-out-of-the-ffmpeg-node/77576/2 "2023-04-15T01:09:38Z")

</div>

> [@Seba3Fi](#):
>
> My current workflow is something like this:  
> -In Scrypted the rstp flows are processed, and motion is detected. This triggers MQTT events that I receive on Node Red. Scrypted also provides some features like exposing to Apple HomeKit and others, and rebroadcasting camera traffic via rtsp.  
> -The second layer is Node Red. Today I am getting a picture from every camera via http, and procesing it with Tensorflow to detect objects of interst.

Just to be clear so that i understand, you have 2 separate systems that are working together. The Scrypted system connects to your rtsp cams and does basic motion detection and notifies your node-red system of the event via mqtt. Then, you want to try to grab the current frame that triggered the motion detection and perform image analysis on it using tensorflow in node-red?

> [@Seba3Fi](#):
>
> Instead of fetching images via http (that provides a lot of timeouts) I would rather use the rtsp stream to get 1 picture at a time via your ffmpeg node.

Are you going to be connecting to the rtsp stream being re-broadcast from Scrypted?

If you are truly going to connect just to grab a single image and then disconnect, i would suggest using the exec node running in exec mode. This will help you to avoid some issues that arise when piping data that is too large to fit inside a single chunk. The exec mode buffers all of the output and gives it all at once when the process exits.

```auto
[{"id":"67d2aab3bceea62c","type":"exec","z":"8b1c500d47fa9005","command":"ffmpeg -i rtsp://somevideostream -f image2pipe -c mjpeg -frames 1 pipe:1","addpay":"","append":"","useSpawn":"false","timer":"","winHide":false,"oldrc":false,"name":"ffmpeg","x":380,"y":700,"wires":[[],[],[]]}]

```

---

<div class="post-metadata">

**Author:** ![zenofmud](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/zenofmud/32/316_2.png) [@zenofmud](https://discourse.nodered.org/u/zenofmud)\
**Post date:** [15 April 2023 01:12 UTC](https://discourse.nodered.org/t/is-there-any-way-to-get-frames-out-of-the-ffmpeg-node/77576/3 "2023-04-15T01:12:09Z")

</div>

A couple of us (@dynamicdave @krambriw and myself) put together a node-red Home Security System. You might find it informative to look as.

> [@Home Surveillance System (HSS) - UPDATE and UPGRADE](https://discourse.nodered.org/t/home-surveillance-system-hss-update-and-upgrade/74699):
>
> UPDATE to HSS (for existing users) David (@dynamicdave), Paul (@zenofmud and Walter (@krambriw) are pleased to announce an update to their Home Surveillance System (HSS) project based on Node-RED running on a Raspberry Pi (recommended hardware is... Raspberry Pi-3B or 4B). Here's a link to the original project-thread. Please note: This update is aimed at existing users who have a system that is up-and-running. This update offers two enhancements... (1) Replaces Single File Gallery (SFG) wi…

---

<div class="post-metadata">

**Author:** ![krambriw](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/krambriw/32/5429_2.png) [@krambriw](https://discourse.nodered.org/u/krambriw)\
**Post date:** [15 April 2023 06:10 UTC](https://discourse.nodered.org/t/is-there-any-way-to-get-frames-out-of-the-ffmpeg-node/77576/4 "2023-04-15T06:10:50Z")

</div>

Hello there, nice project!

When you capture frames triggered by motion detection, maybe you should capture a sequence instead of just one single frame. Otherwise you might miss the actual one of highest interest. Here I think you have to experiment and fine tune according to your needs and camera characteristics

@kevinGodell showed a fine example and in that you can also define how many frames you like to have, like 15  
`ffmpeg -i rtsp://somevideostream -f image2pipe -c mjpeg -frames 15 pipe:1`

With rtsp you also have a delay relative live view but this might in this case be beneficial instead of using http since the frames with relevant motion will "arrive a bit delayed" so the risk of losing the event is less I think & hope. You have to test and try

The teachable machine node looks very interesting, you can train your own models with images from your own cameras. I think such a model should perform better than those generic pre-trained models publicly available. Most of the models around are trained on images with a frontal view of objects but security cameras are normally mounted higher up - looking down. Training a model with images from such mounted cameras should give a good result - in theory. Need to test that at some time with my own images...

Best regards, Walter

---

<div class="post-metadata">

**Author:** ![Seba3Fi](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/seba3fi/32/78891_2.png) [@Seba3Fi](https://discourse.nodered.org/u/Seba3Fi)\
**Post date:** [15 April 2023 10:58 UTC](https://discourse.nodered.org/t/is-there-any-way-to-get-frames-out-of-the-ffmpeg-node/77576/5 "2023-04-15T10:58:22Z")

</div>

> [@krambriw](#):
>
> ffmpeg -i rtsp://somevideostream -f image2pipe -c mjpeg -frames 15 pipe:1

Thanks a lot @kevinGodell , you are 100% right on what I would like to achieve.

The solution proposed works really good, although it takes several seconds to grab an image and return to node-red. I guess it is related to the slow rate of keyframes I have set up on the cameras.  
In order to save processing time to use on tensorflow I had made an attempt to reduce the frame rate to 3, and the 1 keyframe every 4 seconds if I'm not mistaken.  
Lot's of parameters to tweak.  
I have found other threads where you have helped other friends with info, I will take some time to read before asking for more help. Thanks for driving me to this wonderful forum, never been here before, we always learn something new. Take care Kevin!

---

<div class="post-metadata">

**Author:** ![Seba3Fi](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/seba3fi/32/78891_2.png) [@Seba3Fi](https://discourse.nodered.org/u/Seba3Fi)\
**Post date:** [15 April 2023 11:02 UTC](https://discourse.nodered.org/t/is-there-any-way-to-get-frames-out-of-the-ffmpeg-node/77576/6 "2023-04-15T11:02:57Z")

</div>

Thanks a lot @zenofmud, at this stage I am learn what is achievable.

I appreciate the hard work and talent from the members in your project. Definitely we can collaborate and share ideas about features that may be nice to include.

I will buy one of those RASPI4 in the near future.

Take care!

---

<div class="post-metadata">

**Author:** ![Seba3Fi](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/seba3fi/32/78891_2.png) [@Seba3Fi](https://discourse.nodered.org/u/Seba3Fi)\
**Post date:** [15 April 2023 11:07 UTC](https://discourse.nodered.org/t/is-there-any-way-to-get-frames-out-of-the-ffmpeg-node/77576/7 "2023-04-15T11:07:22Z")

</div>

Hi @krambriw, you read my mind. I have in the roadmap to use teachable machines model trained to recognize when garage door is open for instance. This could be exported to scrypted and shown in apple's world as a sensor. Infinite options.  
On grabbing several images at once you are right, my constraint is CPU power, I currently have 4 cameras 2 MegaPix each, and I am planning to add a couple of old phones with rtsp camera app. As a result I came to the conclusion that maybe having a reduced framerate on the cameras and setting 4 seconds as key-frame may help analyze more images.  
Thanks for sharing!

---

<div class="post-metadata">

**Author:** ![kevinGodell](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/kevingodell/32/27040_2.png) [@kevinGodell](https://discourse.nodered.org/u/kevinGodell)\
**Post date:** [15 April 2023 12:00 UTC](https://discourse.nodered.org/t/is-there-any-way-to-get-frames-out-of-the-ffmpeg-node/77576/8 "2023-04-15T12:00:11Z")

</div>

> [@Seba3Fi](#):
>
> Thanks for driving me to this wonderful forum, never been here before, we always learn something new

There are many intelligent and creative people here and there is always somebody willing to help.

> [@Seba3Fi](#):
>
> you are 100% right on what I would like to achieve

I have a follow-up question based on your confirmation.

In Scrypted, are you using the camera's built-in motion detection notification, opencv, or pam-diff?

> [@Seba3Fi](#):
>
> although it takes several seconds to grab an image and return to node-red

A possible workaround would be to maintain a long running rtsp connection from node-red using ffmpeg to the rtsp source at the Scrypted server. You could have it generate a new jpeg once per second and always keep a fresh jpeg available for when you want to send it to the image analysis tensorflow. Of course, it would use more resources, but if you can use hardware acceleration on the node-red box, then the cpu load would be manageable. Personally, I keep a 22 ffmpeg rtsp connections running on my little pi4. Each cam has a main and sub stream. On the 11 sub streams, I decode the h264 encoded video using hardware acceleration to generate 1 jpeg per second so that I can access the jpeg anytime or even stream it as mjpeg, or use it as the poster image for html5 video element. Your mac mini may have hardware accel available to ffmpeg.

If you want to check the ffmpeg, run this command to list the codecs:

```auto
ffmpeg -codecs|grep h264

```

The last line of the output on my pi shows:

```auto
DEV.LS h264 H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 (decoders: h264 h264_v4l2m2m ) (encoders: libx264 libx264rgb h264_omx h264_v4l2m2m h264_vaapi )

```

For the decoder, I use `h264_v4l2m2m` and that ffmpeg process uses a small amount of cpu. FYI, there are limitations on how big the video can be, such as width and height. That's why I use this on my smaller resolution sub streams.

The other possibility is that your camera's substream is outputting mjpeg. In that case, there would be a completely different solution for keeping the jpegs on the ready.

---

<div class="post-metadata">

**Author:** ![Seba3Fi](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/seba3fi/32/78891_2.png) [@Seba3Fi](https://discourse.nodered.org/u/Seba3Fi)\
**Post date:** [15 April 2023 18:23 UTC](https://discourse.nodered.org/t/is-there-any-way-to-get-frames-out-of-the-ffmpeg-node/77576/9 "2023-04-15T18:23:55Z")

</div>

> [@kevinGodell](#):
>
> You could have it generate a new jpeg once per second and always keep a fresh jpeg available for when you want to send it to the image analysis tensorflow. Of course, it would use more resources, but if you can use hardware acceleration on the node-red box, then the cpu load would be manageable. Personally, I keep a 22 ffmpeg rtsp connections running on my little pi4. Each cam has a main and sub stream. On the 11 sub streams, I decode the h264 encoded video using hardware acceleration to generate 1 jpeg per second so that I can access the jpeg anytime or even stream it as mjpeg, or use it as the poster image for html5 video element. Your mac mini may have hardware accel available to ffmpeg.

Thanks a lot @kevinGodell , I do like your idea of having a long lasting connection to ffmpeg and receive the stream at 1 frame per second.

I can actually change the second streams to mpeg and stream to 1 frame per second. Not sure though how to access the info from that stream from node-red and extract that frame every second. I would assume that I could have an HTTP request node requesting the MPEG URL, but from that point on I would have to guess how to extract those frames. Thanks for the idea!

I have run the command on the MAC Mini and here is the output:  
ffmpeg version 5.0.1-tessus [static FFmpeg binaries for macOS 64-bit](https://evermeet.cx/ffmpeg/) Copyright (c) 2000-2022 the FFmpeg developers  
built with Apple clang version 11.0.0 (clang-1100.0.33.17)  
configuration: --cc=/usr/bin/clang --prefix=/opt/ffmpeg --extra-version=tessus --enable-avisynth --enable-fontconfig --enable-gpl --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libfreetype --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libmysofa --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvmaf --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-version3 --pkg-config-flags=--static --disable-ffplay  
libavutil 57. 17.100 / 57. 17.100  
libavcodec 59. 18.100 / 59. 18.100  
libavformat 59. 16.100 / 59. 16.100  
libavdevice 59. 4.100 / 59. 4.100  
libavfilter 8. 24.100 / 8. 24.100  
libswscale 6. 4.100 / 6. 4.100  
libswresample 4. 3.100 / 4. 3.100  
libpostproc 56. 3.100 / 56. 3.100  
DEV.LS h264 H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 (decoders: h264 libopenh264 ) (encoders: libx264 libx264rgb libopenh264 h264\_videotoolbox )

I have read a bit about ffmpeg but not certain which one would help using MAC Mini's GPU.

I am sure scrypted and node-red could run on its own hardware, but right now everything runs on my old MAC Mini.

Regarding hardware acceleration I need to buy one of those TPUs, but still don't have it (those come with 100% or more taxes in Brazil where I am living right now)

Thanks,  
Sebastián

---

<div class="post-metadata">

**Author:** ![krambriw](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/krambriw/32/5429_2.png) [@krambriw](https://discourse.nodered.org/u/krambriw)\
**Post date:** [16 April 2023 05:07 UTC](https://discourse.nodered.org/t/is-there-any-way-to-get-frames-out-of-the-ffmpeg-node/77576/10 "2023-04-16T05:07:20Z")

</div>

> [@Seba3Fi](#):
>
> I can actually change the second streams to mpeg and stream to 1 frame per second. Not sure though how to access the info from that stream from node-red and extract that frame every second. I would assume that I could have an HTTP request node requesting the MPEG URL, but from that point on I would have to guess how to extract those frames. Thanks for the idea!

> **[node-red-contrib-multipart-stream-decoder](https://flows.nodered.org/node/node-red-contrib-multipart-stream-decoder)**
>
> Node Red node for decoding multipart streams over http

---

<div class="post-metadata">

**Author:** ![Seba3Fi](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/seba3fi/32/78891_2.png) [@Seba3Fi](https://discourse.nodered.org/u/Seba3Fi)\
**Post date:** [16 April 2023 11:23 UTC](https://discourse.nodered.org/t/is-there-any-way-to-get-frames-out-of-the-ffmpeg-node/77576/11 "2023-04-16T11:23:03Z")

</div>

Wonderful @krambriw this is going to make it much easier. Thanks!

---

<div class="post-metadata">

**Author:** ![Seba3Fi](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/seba3fi/32/78891_2.png) [@Seba3Fi](https://discourse.nodered.org/u/Seba3Fi)\
**Post date:** [16 April 2023 21:25 UTC](https://discourse.nodered.org/t/is-there-any-way-to-get-frames-out-of-the-ffmpeg-node/77576/12 "2023-04-16T21:25:11Z")

</div>

Thanks for the suggestion @krambriw , now my flow is implemented on "node-red-contrib-multipart-stream-decoder".

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/f/1/f17fdc656f0e3aac126ed528a96e9d12fc454878.png)

With this flow I am able to process 1 FPS for 4 cameras in a sustained timeframe, which is enaugh by now.

My next challenge is to get the motion detection to work. It has stopped working on srypted, they update stuff very often 😉 They use OPENCV to detect motion, and my old MAC Mini is not so friendly with that.

My plan B then is detecting motion on node-red, I have found a node "node-red-contrib-camera-motion" that works fine when there is only one instance, but behaives wierdly when more than one instance is in place.

Plan C is building the motion detector with [node-red-contrib-image-tools (node) - Node-RED](https://flows.nodered.org/node/node-red-contrib-image-tools), they have very interesting modules too.

Side note, some years ago I had tweaked some OPENCV/Python code that worked really fine on Windows.

Take Care

---

<div class="post-metadata">

**Author:** ![krambriw](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/krambriw/32/5429_2.png) [@krambriw](https://discourse.nodered.org/u/krambriw)\
**Post date:** [17 April 2023 05:07 UTC](https://discourse.nodered.org/t/is-there-any-way-to-get-frames-out-of-the-ffmpeg-node/77576/13 "2023-04-17T05:07:46Z")

</div>

It is thanks to @BartButenaers we have the "node-red-contrib-multipart-stream-decoder" node, they work also very well, I use it myself in some applications. For http streams it is great!

I briefly looked at Scrypted, looks very nice and modern! Strange if they would remove such an important feature as motion detection. MAybe is on your side, you need to update OpenCV?? I don't know

Motion detection normally starts with comparing frames, number of pixel changes, but it is so much more. It has to handle many things like fast light changes, trees moving in the wind, supporting masks and much more

Alternatives to Scrypted? Maybe you could check out Shinobi that @kevinGodell also works/worked with but I do not know if it meets all your requirements and how it can integrate with Node-RED

Myself I use the pretty old but still maintained Motion, it works also very well as "middleware & motion detector" with my usb cameras. My "residential" setup is pretty customized to my needs:

- usb cameras are connected to a number of distributed RPi's, all running Node-RED and Motion
- when motion is detected, local recording starts and frames are sent for analyze via MQTT
- the analyzer is a Python script with YOLO v4 for object detection, running on a NVIDIA Jetson Nano
- if selected objects are found, such images are stored and notifications sent via Telegram
- live viewing of cameras, stored images from analyzes, playback of recordings is all handled by Node-RED and it's dashboard running on the same Jetson Nano

Object detection using YOLO is very accurate and resource intensive. That's why the Jetson Nano so YOLO can utilize the power of GPU's. Analyzing an image takes in my case some 0.3 seconds. Running the same script in a RPi4, the analyze takes almost 10 seconds

Best regards, Walter

---

<div class="post-metadata">

**Author:** ![Seba3Fi](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/seba3fi/32/78891_2.png) [@Seba3Fi](https://discourse.nodered.org/u/Seba3Fi)\
**Post date:** [18 April 2023 20:10 UTC](https://discourse.nodered.org/t/is-there-any-way-to-get-frames-out-of-the-ffmpeg-node/77576/14 "2023-04-18T20:10:44Z")

</div>

> [@krambriw](#):
>
> Object detection using YOLO is very accurate and resource intensive. That's why the Jetson Nano so YOLO can utilize the power of GPU's. Analyzing an image takes in my case some 0.3 seconds. Running the same script in a RPi4, the analyze takes almost 10 seconds

Thanks a lot @krambriw , we have different setups, but maintaining the main ideas.  
During the week I have very little time to spend on Node-Red, but during the afternoon I will catch-up.

Take care,  
Sebastián

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [17 June 2023 20:11 UTC](https://discourse.nodered.org/t/is-there-any-way-to-get-frames-out-of-the-ffmpeg-node/77576/15 "2023-06-17T20:11:41Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.