[beta testing] nodes for live streaming mp4

I digress a bit from the subject: I did not find the command to update from Stretch to Buster. Do i have to completely re-flash the SD card?

[EDIT] ifound https://www.raspberrypi.org/blog/buster-the-new-version-of-raspbian/

see - https://www.makeuseof.com/tag/raspberry-pi-update-raspbian-os/

1 Like

My advice, SD cards are pretty cheap these days, start fresh with a new card, then you can return to your original system if necessary.

I'm usually grateful to have had the old system available, as I seem to always forget something, being able to go back at look at what I changed in /etc or retrieve something from /home/pi has always more than justified the cost of a fresh SD card.

2 Likes

I have indeed upgraded several RPi3's from Stretch to Buster succesfully following the steps listed in the article linked to by @dceejay. It has worked all the time (I even upgraded a few from Jessie -> Stretch -> Buster following the same principles).

However, I really recommend to make a backup of the SD card before moving on. In all my RPi's I do have a permanent connected microSDcard reader with an additional SD card inserted. When I make a "major" change of something, I always run the easy to use "SD Card Copier" tool. This will save my hard work the day an active SD card eventually would die (I have also tested this, it creates a perfect clone, swapping the SD cards works fine, the RPi boots up and runs exactly as expected with the clone inserted)

EDIT: Also checked one of my upgraded RPi's, for sure ffmpeg ver 4 was there, it's not the latest but "good enough":

ffmpeg version 4.1.4-1+rpt7~deb10u1 Copyright (c) 2000-2019 the FFmpeg developers

  libavutil      56. 22.100 / 56. 22.100
  libavcodec     58. 35.100 / 58. 35.100
  libavformat    58. 20.100 / 58. 20.100
  libavdevice    58.  5.100 / 58.  5.100
  libavfilter     7. 40.101 /  7. 40.101
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  3.100 /  5.  3.100
  libswresample   3.  3.100 /  3.  3.100
  libpostproc    55.  3.100 / 55.  3.100
Muxer mp4 [MP4 (MPEG-4 Part 14)]:
    Common extensions: mp4.
    Mime type: video/mp4.
    Default video codec: h264.
    Default audio codec: aac.
mp4 muxer AVOptions:
  -movflags          <flags>      E........ MOV muxer flags (default 0)
     rtphint                      E........ Add RTP hint tracks
     empty_moov                   E........ Make the initial moov atom empty
     frag_keyframe                E........ Fragment at video keyframes
     frag_every_frame              E........ Fragment at every frame
     separate_moof                E........ Write separate moof/mdat atoms for e

In one of my RPi's I decided to build from source so here I have a newer version (but I guess both would work)

ffmpeg version N-99578-gaf701196ec Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 8 (Raspbian 8.3.0-6+rpi1)
  configuration: --extra-ldflags=-latomic --arch=armel --target-os=linux --enable-gpl --enable-omx --enable-omx-rpi --enable-libx264 --enable-nonfree
  libavutil      56. 60.100 / 56. 60.100
  libavcodec     58.111.101 / 58.111.101
  libavformat    58. 62.100 / 58. 62.100
  libavdevice    58. 11.102 / 58. 11.102
  libavfilter     7. 87.100 /  7. 87.100
  libswscale      5.  8.100 /  5.  8.100
  libswresample   3.  8.100 /  3.  8.100
  libpostproc    55.  8.100 / 55.  8.100
Muxer mp4 [MP4 (MPEG-4 Part 14)]:
    Common extensions: mp4.
    Mime type: video/mp4.
    Default video codec: h264.
    Default audio codec: aac.
mp4 muxer AVOptions:
  -movflags          <flags>      E......... MOV muxer flags (default 0)
     rtphint                      E......... Add RTP hint tracks
     empty_moov                   E......... Make the initial moov atom empty
     frag_keyframe                E......... Fragment at video keyframes
     frag_every_frame              E......... Fragment at every frame
     separate_moof                E......... Write separate moof/mdat atoms for 

@dceejay @wb666greene @krambriw what happened. RPI down! I was optimistic :grin:

ok i got it : frag_every_frame is here after Buster update :slightly_smiling_face:

After a lot of fright I updated my RPI3B with Buster (thanks to @dceejay ), ffmpeg is on version 4, and I can use your new command of "frag_every_frame" and the result is perfect:
image

UNLESS I lose the signal, I lose 7 seconds which will not be caught automatically.
image

I indicate that the camera is in wifi, where it is I cannot pull an ethernet cable.
Losing the connection for a few seconds is not dramatic, the ideal would be to speed up the video to find real time. Too bad for the fluidity if it happens a few times a day it goes unnoticed.

2 Likes

I do not understand this fully, just for my understanding:

  • the camera connection is lost, say 7 seconds
  • then connection is back and streaming works again
  • time showed is 7 seconds "behind"

Are you sure also video is behind and not only showed time is "wrong" ??

I mean, if the camera connection is lost, there is not possible that frames are transferred and shown later as "behind" or delayed?? Maybe is the node that is "stopping" time counting and when frames are coming again, continuos showing time from when frames stopped arriving?? Therefore a 7 second lost connection will show as 7 second behind? But this is just a guess

My reference stream is the one played by VLC because it is the one that plays the video without any delay compared to the original streamming of the IP camera (Android app, CMS, IP camera client on browser).

So, if I put VLC side by side and the UI_MP4FRAG (of our friend Kevin :wink:) , when I lose the signal (weak wifi):

  • VLC stops a few seconds and accelerates the video to resume real time
  • UI_MP4FRAG "turns in circles" waiting for the return of the signal, then resume reading but late compared to VLC.

In this example I am in the worst case, in FHD so the wifi signal is in difficulty and yes, the accumulated delay went up to 7s "behind" compared to VLC.

Aha, understood, so the acceleation is the point (or simply skipping frames to start just to catch up with "live video"??)

Since the goal is to be the least late compared to the original flow.

Was this using the new ffmpeg movflag frag_every_frame? I have noticed an issue with atleast 1 of my cams that generally performs poorly and can only generate 10s duration segments based on the iframe interval. It can take up to 10s for the video to start being rendered after receiving segments, since I suspect that some of the newly received (shortened) segments don't have enough data to complete the visual frame.

Another thing I am currently working on is a different way to push the segments from server to client when using socket io. Currently, each new segment is requested after receiving the previous segment. I am going to add the option to request "all" segments, so there only needs to be the single request and all new segments will be sent as soon as they are available from the server. Not sure if this will make a difference or not, but it might take a small burden off of the socket io server.

I could implement another setting on the ui that can detect and fast forward the currentTime property of the video element. The tricky part is knowing what is a good value. I can constantly check it and see the current time vs available time and move the play head forward. The downside to that is if you move too far forward and dont receive a new video segment in time, then you will see lots of the spinning loading icon since the player will be stalled waiting on new video content.

1 Like

Yes i applied this new change command , with the ffmpeg v4 :

:thinking: yes I see it is difficult to find the happy medium.

Would you not be able to calculate a kind of "moving average segment rate" that can be used as a "move forward" stepper

Did you create that nice UI with node-red dashboard ?

[off topic] Yes everything is done with the Node-red Dashboard, with some Dashboard ui-template keys (except the wifi chart which is a Win10 window in the background).

  • a home page with tabs of Functions to develop (or not). Each tab has a button that leads to a more complete page.
  • each complete page contains a button to return to the home page.
    image
    it looks something like this : ui control is the conductor of all navigation
2 Likes

Made some changes to dynamically tweak the video's currentTime property to keep it at a reasonable position based on the current amount of buffered video in an attempt to keep it close to realtime. This will eventually be a setting that you can change.

Also, found a major flaw that caused the video to freeze if a segment failed to arrive and the source buffer ended up with a gap in the time ranges.

If you wish to use the shortened segments with the -movflag +frag_every_frame and -min_frag_duration while still support hls.js playback, then you will have to tweak some of its settings to handled the smaller segments. An hls.js configuration that worked for me when my segments were shortened to 0.5 second duration:

{
    "liveDurationInfinity": true,
    "liveBackBufferLength": 5,
    "maxBufferLength": 10,
    "manifestLoadingTimeOut": 1000,
    "manifestLoadingMaxRetry": 10,
    "manifestLoadingRetryDelay": 500
}

I am currently testing socket.io@3 as a dependency. For me it seems to have better performance. If you would like to see if we can make this work and it doesn't break your node red server, here is what you can try:

// change to the node red folder
cd .node-red

// uninstall the dashboard and mp4frag nodes
npm uninstall node-red-dashboard
npm uninstall node-red-contrib-mp4frag
npm uninstall node-red-contrib-ui-mp4frag

// install the dashboard and mp4frag nodes that use socket.io@3
npm install kevinGodell/node-red-dashboard
npm install kevinGodell/node-red-contrib-mp4frag#socket_io_3x
npm install kevinGodell/node-red-contrib-ui-mp4frag#socket_io_3x

// restart node red
node-red-stop && node-red-start

Updated accordingly

But if I set socket.io as nbr 1 choice in Player Order (in ui-mp4-frag) nothing is shown in the browser. On my Mac the following works:
Chrome: hls.js, hls, mp4
Safari: hls.js, hls

Thanks for testing and I am sorry that it did not work for you. I am investigating now. If you have a chance, I am curious if you have any other dependencies on socket io. Can you also show the output of the command npm ls socket.io in the .node-red folder ?

pi@raspberrypi:~ $ cd .node-red
pi@raspberrypi:~/.node-red $ npm ls socket.io
node-red-project@0.0.1 /home/pi/.node-red
├─┬ node-red-contrib-mp4frag@0.1.0 (github:kevinGodell/node-red-contrib-mp4frag#9b46fcc281cb15c0f48219b211cce4e3e61dff31)
│ └── socket.io@3.0.3 
└─┬ node-red-dashboard@2.24.2-beta (github:kevinGodell/node-red-dashboard#7ba4a3c36f034960959c119c50fcf3b4fdd0d964)
  └── socket.io@3.0.3  deduped

The socket.io.js file delivered from the node-red server should show the version at the top. The file at my server was found at http://192.168.1.85:1880/ui/socket.io/socket.io.js.

/*!
 * Socket.IO v3.0.3
 * (c) 2014-2020 Guillermo Rauch
 * Released under the MIT License.
 */

And in the browser, the socket connection should show that it is using EIO version 4, which is what socket io v3 uses.

Request URL:
ws://192.168.1.85:1880/mp4frag/socket.io/?EIO=4&transport=websocket

Request URL:
ws://192.168.1.85:1880/ui/socket.io/?EIO=4&transport=websocket&sid=6crnnvg2J0IMM1unAABW