Media source extension video player for live streaming mp4

[kevinGodell]

I am in need of development recommendations for making a node video player based on media source extensions (similar to what hls.js uses).

What I know:

• I can make a very basic node, e.g. node-red-contrib-pipe2jpeg
• I have already made a video player (using media source extensions) years ago for live streaming mp4 with nodejs as my backend

What I plan to do:

• use ffmpeg to connect to rtsp ip cam [✓]
• stream copy mp4 video from the source [✓]
• make an mp4 parser node (to get the mime, init, and segments) [✓]
• keep everything in memory only (no writing files) [✓]
• make simple no frills html5 video player (based on media source extensions) [ ]
• send the mp4 chunks to the video player node (websockets) [ ]

What I don't know:

• delivering my mp4 chunks from source node to the video player node

I am not sure what is already built-in to node-red. I am not sure how to connect a node that has mp4 chunks to a node that can play the mp4 chunks. I prefer to use websockets. I don't know if node-red uses some magic to automatically connect nodes like this or if i have to create my own websocket server, etc. Any recommendations would be appreciated.

[Nxito]

I did not clearly understand the question, but I did something similar to this you comment. To transmit ffmpeg through websocket to an html page I use jsmpeg, I don't know if it will help you

phoboslab/jsmpeg

[kevinGodell]

Thanks for the response and the link. I can make the video player. I am just not sure how to connect nodes when delivering back end content to a front end video player.
So, this is really a question about developing for node-red, not about making a video player. Sorry my unclear post.

[BartButenaers]

Hi Kevin,

@kevinGodell

for making a node video player based on media source extensions (similar to what hls.js uses).

A few questions about this:

1. Is there any advantage of building a video player yourself, instead of using an existing one. I have been experimenting a few weeks ago with video.js in the dashboard. They have quite some plugins already, e.g. for live streaming. And the user interface is quite customizable: I have e.g. been playing with a svg overlay on top of it (e.g. to show leds for recording, and other kind of stuff). Reason why I wanted to go that way, is because I experimented already some time ago to build my own node-red-contrib-ui-camera-viewer. But it is quite some work if you need to build it from scratch.

   Any ideas about this??

2. I assume you want to build a UI node so we can use it in the Node-RED dashboard?

@kevinGodell

I don't know if node-red uses some magic to automatically connect nodes like this or if i have to create my own websocket server, etc.

If you build a UI node for the Node-RED dashboard, then the whole websocket stuff is already in place for you. And the dashboard offers functions that you can use to intercept and manipulate data to be send.
However I experienced already many times that pushing large amounts of data through the websocket channel will make your UI unresponsive. On the other hand when you - instead of pushing through the websocket channel - let the UI pull the video chunks then it works much smoother...
But - based on your large experience in the area - I assume that you have good experiences with websockets for mp4 chunks in other applications??

[kevinGodell]

Is there any advantage of building a video player yourself, instead of using an existing one. I have been experimenting a few weeks ago with video.js in the dashboard. They have quite some plugins already, e.g. for live streaming. And the user interface is quite customizable: I have e.g. been playing with a svg overlay on top of it (e.g. to show leds for recording, and other kind of stuff). Reason why I wanted to go that way, is because I experimented already some time ago to build my own node-red-contrib-ui-camera-viewer. But it is quite some work if you need to build it from scratch.

The main advantage is that it is simple. The popular video players have too many features that are not needed. We already have the playable mp4 chunks, just have to feed them to media source player to get video. No complicated configurations needed or writing files to disk or multi bitrate streams or rewinding. Just (near) live video.

I assume you want to build a UI node so we can use it in the Node-RED dashboard?

yes

If you build a UI node for the Node-RED dashboard, then the whole websocket stuff is already in place for you. And the dashboard offers functions that you can use to intercept and manipulate data to be send.

Are you saying that if i wire a server-side node to a UI node, that the data will be sent without me needing to create my own websocket or http server? If yes, then that is the info I am looking for. I need to get a hold of a hollowed out simplistic template of a UI node that can receive data via the msg.payload.

However I experienced already many times that pushing large amounts of data through the websocket channel will make your UI unresponsive. On the other hand when you - instead of pushing through the websocket channel - let the UI pull the video chunks then it works much smoother...

Without me knowing the finer details about your situation, perhaps it was due to sending 5 jpegs per second or something similar? Sending mp4 chunks should use less bandwidth and will at the fastest rate be sent maybe every 2 seconds, which is usually the smallest chunk duration based on my experiments with rtsp cams.

But - based on your large experience in the area - I assume that you have good experiences with websockets for mp4 chunks in other applications??

My personal cctv system at my house uses socket.io to deliver mp4 and jpeg for 14 cams. Also, a variation of my video player is used on a very popular nodejs based cctv solution.

[BartButenaers]

@kevinGodell

The main advantage is that it is simple... Just (near) live video.

Ok. From what I remember - apart from the mp4 complexity itself of course - the main problem was to get my SVG drawing correctly on top of the video. When you look at the first screenshot in this discussion, you will see exactly why I loved to have that so much!!

@kevinGodell

Are you saying that if i wire a server-side node to a UI node, that the data will be sent without me needing to create my own websocket or http server? If yes, then that is the info I am looking for. I need to get a hold of a hollowed out simplistic template of a UI node that can receive data via the msg.payload.

Yes it is out-of-the-box functionality. If you want to share your media source code, I can try to integrate it into my camera-viewer UI node. It is your call ...

@kevinGodell

My personal cctv system at my house uses socket.io to deliver mp4 and jpeg for 14 cams.

That is so weird. I had severe problems when displaying multiple camera images at once. That is the reason why I developed the node-red-contrib-multipart-stream-encoder node at the time being: when you pull data simultaneously for N camera's that works fine, but if you push it the whole dashboard became unresponsive...

The Node-RED dashboard also uses socket.io as you can see here in the code. So that is good news I assume, because it should be able to work without getting unresponsive. Since it works outside of Node-RED, it might be useful that we start analyzing the problem... Because pushing allows us to keep the flows very simple, compared to pulling.

[ristomatti]

@kevinGodell

Also, a variation of my video player is used on a very popular nodejs based cctv solution.

Shinobi I assume? Very fascinating topic, I hope you guys can get this working and will later share the results!

[kevinGodell]

I will give you an insight, since you assumed correctly. It was moe at shinobi that inspired me to start developing again, after taking many years off (just a hobby). I found him online mentioning an alternative to zoneminder (resource hog), which I was currently using. I had not even heard of nodejs at the time. When I found out it was javasript running server side, I felt a little sick. But after playing with it, I fell in love with nodejs and can't get enough of it.

[ristomatti]

@kevinGodell

When I found out it was javasript running server side, I felt a little sick. But after playing with it, I fell in love with nodejs and can't get enough of it.

My exact reaction when I first heard about a JS backend. The reaction was unwarranted and wrong indeed.

[kevinGodell]

I have made some progress for the back-end mp4 stuff, although it is slow going since I have very little time or knowledge of node-red.

For regular http requests, I am currently able to create a live streaming source that can be played natively in the safari browser of mac/ios by pointing to the playlist.m3u8. This should also be able to be played in other browsers(supporting media source extensions) by hls.js or the desktop vlc app or any other player that can consume m3u8. This involved some simple request routes and utilizing the global context.

Have not touched anything related to sockets yet, although it should not be impossible with the mp4 data being accessible in the global context.

I think the overall idea is that my node will just be an accessible mp4 source and anybody can interact how they need using it various end points.

I will try to clean it up and publish it and share a flow soon(day or 2) if anybody is interested in trying it out.

[BartButenaers]

@kevinGodell

although it is slow going since I have very little time or knowledge of node-red

You know where to find me... But as I already said, it is your call...

@kevinGodell

and publish it

It is also useful to discuss an alpha version of your nodes here, before you publish a final version on npm! Most of the time you get good feedback. But once published a final version on npm, it is difficult to make design changes without breaking existing flows...

[dceejay]

echoing @BartButenaers - yes - always best to get feedback from an alpha before publishing. Most folk here are happy to pull direct from github for testing.

[kevinGodell]

Thanks @BartButenaers @dceejay. I appreciate the support. I am almost ready to share a basic version of the mp4frag backend that will generate a live m3u8 playlist that can be played by hls.js or natively on Safari Mac/iOS.

The current problem I am having is making a ui template that includes the sample code for hls.js video player example. I had 2 nodes, 1 to load hls.js in the head, and the other to load the video player. It was hit or miss with half the time HLS giving an error about being undefined. Can you show me a flow for a simple hls.js or video.js video player than can consume a m3u8 playlist? I am not very good with setting up node-red configurations yet.

I will be back at my computer in about 8 hours and I will try to push some code.

[BartButenaers]

Kevin,
I will create a basic ui node, based on hls.js. But my holidays are over since yesterday, so it will take a bit longer.
Then we can keep adding features to that node in the near future.
Bart

[BartButenaers]

I see here that hls.js is not supported on all browsers. So I'm wonderung whether we should call it node-red-contrib-ui-hls, or node-red-contrib-ui-.... in case we need to support - beside hls - other types of streaming afterwards? Any thoughts,

[kevinGodell]

I think Hls.js is supported by all modern browsers that support media source extensions, and it can fallback to native hls for safari. Personally, I only support modern browsers.

The video player can be somewhat neutral with the library it uses, such as hls.js for http Or you can opt to use Websocket/socket.io to feed the mp4 fragments instead of http.

It can be more generic as a general live mp4 player.

[BartButenaers]

@kevinGodell

I am almost ready to share a basic version of the mp4frag backend that will generate a live m3u8 playlist

That is fantastic news !!!

@kevinGodell

Personally, I only support modern browsers

Deal...

@kevinGodell

hls.js for http Or you can opt to use Websocket/socket.io to

So I assume that node-red-contrib-ui-mp4-player might be a better name then.

[BartButenaers]

Hey Kevin,

I have quickly create a minimal HLS player for the dashboard: node-red-contrib-ui-mp4-player

I have you added already as contributor in the package.json file and in the copyright headers of the js and html files. Will try to remind tomorrow that I also send you a Github invitation to cooperate.

You can install it directly from my Github account like this:

npm install bartbutenaers/node-red-contrib-ui-mp4-player

It contains a minimal config screen at the moment:

• The source can be an URL (to fetch) or an input msg field (to push). Currently only the URL is implemented!!
• The aspect ratio field doesn't work for some reason. Will need to look at that later ...

And when you open the dashboard, it will automatically start playing the m3u8 link:

Tomorrow evening I won't have much time, but then at least we have something to discuss and start testing.

Like always all 'constructive' feedback from anybody is very welcome!!

Have fun with HLS

Bart

[kevinGodell]

@BartButenaers Your video player works great. Thanks. It really helps for me to test my back end much easier.

Here is an example flow using your player with my mp4frag wrapper using external routes with mp4frag being added to global context. I started with the context approach to be more flexible in case somebody wanted to build something around the lib (and at the time i didnt know i could make my own routes )

[{"id":"26b5a4eb.db1d34","type":"tab","label":"Flow 1","disabled":false,"info":""},{"id":"95acc154.09f0c","type":"inject","z":"26b5a4eb.db1d34","name":"Start stream","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"true","payloadType":"str","x":110,"y":40,"wires":[["7dbc80c7.4e764"]]},{"id":"8b7657be.b17558","type":"inject","z":"26b5a4eb.db1d34","name":"Stop stream","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"false","payloadType":"str","x":110,"y":86,"wires":[["7dbc80c7.4e764"]]},{"id":"7dbc80c7.4e764","type":"switch","z":"26b5a4eb.db1d34","name":"","property":"payload","propertyType":"msg","rules":[{"t":"eq","v":"true","vt":"str"},{"t":"eq","v":"false","vt":"str"}],"checkall":"true","repair":false,"outputs":2,"x":261,"y":40,"wires":[["70626fe2.0e9c5"],["fd61d017.59965"]]},{"id":"fd61d017.59965","type":"function","z":"26b5a4eb.db1d34","name":"stop","func":"msg= [\n    {\n    kill:'SIGHUP',\n    payload : 'SIGHUP'\n    }\n    \n    \n    ];     // set a new payload & the counter\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":281,"y":89,"wires":[["70626fe2.0e9c5"]]},{"id":"70626fe2.0e9c5","type":"exec","z":"26b5a4eb.db1d34","command":"ffmpeg -loglevel quiet -hwaccel rpi -c:v h264_mmal -rtsp_transport tcp -i  rtsp://192.168.1.4:554/user=admin_password=pass_channel=0_stream=0.sdp?real_stream -q 2 -vf fps=fps=2,scale=-1:-1 -c:v mjpeg -f image2pipe pipe:2 -an -c:v copy -f mp4 -movflags +frag_keyframe+empty_moov+default_base_moof pipe:1","addpay":false,"append":"","useSpawn":"true","timer":"","oldrc":false,"name":"front porch ip cam main","x":474,"y":49,"wires":[["9bfbb50a.54833"],["752eb.8f4e1d158"],["9bfbb50a.54833"]]},{"id":"522ad11.3d8643","type":"image","z":"26b5a4eb.db1d34","name":"","width":"320","data":"payload","dataType":"msg","thumbnail":false,"active":true,"pass":false,"outputs":0,"x":140,"y":180,"wires":[]},{"id":"752eb.8f4e1d158","type":"pipe2jpeg","z":"26b5a4eb.db1d34","name":"some name","x":680,"y":80,"wires":[["522ad11.3d8643"]]},{"id":"f94373f8.d3d9b","type":"http in","z":"26b5a4eb.db1d34","name":"","url":":base([a-z_]+).m3u8","method":"get","upload":false,"swaggerDoc":"","x":521,"y":196,"wires":[["575c6379.bf74dc"]]},{"id":"d360f454.75f2b8","type":"http in","z":"26b5a4eb.db1d34","name":"","url":"init-:base([a-z_]+).mp4","method":"get","upload":false,"swaggerDoc":"","x":531,"y":236,"wires":[["ccdad96e.857a08"]]},{"id":"bc4e092d.1ee938","type":"http in","z":"26b5a4eb.db1d34","name":"","url":":base([a-z_]+):seq(\\d+).m4s","method":"get","upload":false,"swaggerDoc":"","x":541,"y":276,"wires":[["39c27647.a5d5ca"]]},{"id":"be9c49a3.a66a78","type":"http in","z":"26b5a4eb.db1d34","name":"","url":":base([a-z_]+).m3u8.txt","method":"get","upload":false,"swaggerDoc":"","x":531,"y":316,"wires":[["3827e276.b31f8e"]]},{"id":"3827e276.b31f8e","type":"function","z":"26b5a4eb.db1d34","name":"mp4frag context","func":"const { base } = msg.req.params;\n\nconst mp4frag = global.get(base);\n\nconst m3u8 = mp4frag && mp4frag.m3u8;\n\nif (m3u8) {\n    msg.payload = m3u8;\n    msg.headers = {'content-type': 'text/plain'};\n} else {\n    msg.payload = 'm3u8 playlist not found';\n    msg.headers = {'content-type': 'text/plain'};\n    msg.statusCode = 404;\n}\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":791,"y":316,"wires":[["228a5549.40f13a"]]},{"id":"228a5549.40f13a","type":"http response","z":"26b5a4eb.db1d34","name":"","statusCode":"","headers":{},"x":961,"y":316,"wires":[]},{"id":"39c27647.a5d5ca","type":"function","z":"26b5a4eb.db1d34","name":"mp4frag context","func":"const { base, seq } = msg.req.params;\n\nconst mp4frag = global.get(base);\n\nconst segment = mp4frag && mp4frag.getHlsSegment(seq);\n\nif (segment) {\n    msg.payload = segment;\n    msg.headers = {'content-type': 'video/mp4'};\n} else {\n    msg.payload = `segment ${seq} not found`;\n    msg.headers = {'content-type': 'text/plain'};\n    msg.statusCode = 404;\n}\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":790,"y":276,"wires":[["4231b8b3.b0fa68"]]},{"id":"4231b8b3.b0fa68","type":"http response","z":"26b5a4eb.db1d34","name":"","statusCode":"","headers":{},"x":961,"y":276,"wires":[]},{"id":"ccdad96e.857a08","type":"function","z":"26b5a4eb.db1d34","name":"mp4frag context","func":"const { base } = msg.req.params;\n\nconst mp4frag = global.get(base);\n\nconst initialization = mp4frag && mp4frag.initialization;\n\nif (initialization) {\n    msg.payload = initialization;\n    msg.headers = {'content-type': 'video/mp4'};\n} else {\n    msg.payload = 'initialization fragment not found';\n    msg.headers = {'content-type': 'text/plain'};\n    msg.statusCode = 404;\n}\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":788,"y":236,"wires":[["270da754.a5f5e8"]]},{"id":"270da754.a5f5e8","type":"http response","z":"26b5a4eb.db1d34","name":"","statusCode":"","headers":{},"x":961,"y":236,"wires":[]},{"id":"575c6379.bf74dc","type":"function","z":"26b5a4eb.db1d34","name":"mp4frag context","func":"const { base } = msg.req.params;\n\nconst mp4frag = global.get(base);\n\nconst m3u8 = mp4frag && mp4frag.m3u8;\n\nif (m3u8) {\n    msg.payload = m3u8;\n    msg.headers = {'content-type': 'application/vnd.apple.mpegURL'};\n    //msg.headers = {'content-type': 'application/x-mpegURL'};\n    //msg.headers = {'content-type': 'application/x-mpegurl'};\n} else {\n    msg.payload = 'm3u8 playlist not found';\n    msg.headers = {'content-type': 'text/plain'};\n    msg.statusCode = 404;\n}\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":787,"y":196,"wires":[["5e29cec.60df53"]]},{"id":"5e29cec.60df53","type":"http response","z":"26b5a4eb.db1d34","name":"","statusCode":"","headers":{},"x":961,"y":196,"wires":[]},{"id":"9bfbb50a.54833","type":"mp4frag","z":"26b5a4eb.db1d34","uniqueName":"front_porch_main","hlsListSize":"4","contextAccess":"global","httpRoutes":false,"x":710,"y":20,"wires":[[]]},{"id":"387294df.f4c8f4","type":"ui_mp4_player","z":"26b5a4eb.db1d34","group":"ea204528.5e15d8","order":4,"width":"6","height":"3","name":"","sourceType":"url","sourceValue":"http://192.168.1.85:1880/front_porch_main.m3u8","aspectratio":"stretch","x":910,"y":60,"wires":[[]]},{"id":"ea204528.5e15d8","type":"ui_group","name":"Group 1","tab":"ef1a3eec.2694d","order":1,"disp":true,"width":6},{"id":"ef1a3eec.2694d","type":"ui_tab","z":"26b5a4eb.db1d34","name":"Dashboard","icon":"dashboard","order":1,"disabled":false,"hidden":false}]

npm install kevinGodell/node-red-contrib-mp4frag

I am going to push some code in a little while to add the built in http routes and then will add another flow showing the alternative approach.

[kevinGodell]

Updated the code to use internal http routes instead of needing global context and external routes:

[{"id":"26b5a4eb.db1d34","type":"tab","label":"Flow 1","disabled":false,"info":""},{"id":"95acc154.09f0c","type":"inject","z":"26b5a4eb.db1d34","name":"Start stream","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"true","payloadType":"str","x":110,"y":40,"wires":[["7dbc80c7.4e764"]]},{"id":"8b7657be.b17558","type":"inject","z":"26b5a4eb.db1d34","name":"Stop stream","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"false","payloadType":"str","x":110,"y":86,"wires":[["7dbc80c7.4e764"]]},{"id":"7dbc80c7.4e764","type":"switch","z":"26b5a4eb.db1d34","name":"","property":"payload","propertyType":"msg","rules":[{"t":"eq","v":"true","vt":"str"},{"t":"eq","v":"false","vt":"str"}],"checkall":"true","repair":false,"outputs":2,"x":261,"y":40,"wires":[["70626fe2.0e9c5"],["fd61d017.59965"]]},{"id":"fd61d017.59965","type":"function","z":"26b5a4eb.db1d34","name":"stop","func":"msg= [\n    {\n    kill:'SIGHUP',\n    payload : 'SIGHUP'\n    }\n    \n    \n    ];     // set a new payload & the counter\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":281,"y":89,"wires":[["70626fe2.0e9c5"]]},{"id":"70626fe2.0e9c5","type":"exec","z":"26b5a4eb.db1d34","command":"ffmpeg -loglevel quiet -hwaccel rpi -c:v h264_mmal -rtsp_transport tcp -i  rtsp://192.168.1.4:554/user=admin_password=pass_channel=0_stream=0.sdp?real_stream -q 2 -vf fps=fps=2,scale=-1:-1 -c:v mjpeg -f image2pipe pipe:2 -an -c:v copy -f mp4 -movflags +frag_keyframe+empty_moov+default_base_moof pipe:1","addpay":false,"append":"","useSpawn":"true","timer":"","oldrc":false,"name":"front porch ip cam main","x":474,"y":49,"wires":[["9bfbb50a.54833"],["752eb.8f4e1d158"],["9bfbb50a.54833"]]},{"id":"522ad11.3d8643","type":"image","z":"26b5a4eb.db1d34","name":"","width":"320","data":"payload","dataType":"msg","thumbnail":false,"active":true,"pass":false,"outputs":0,"x":140,"y":180,"wires":[]},{"id":"752eb.8f4e1d158","type":"pipe2jpeg","z":"26b5a4eb.db1d34","name":"some name","x":680,"y":80,"wires":[["522ad11.3d8643"]]},{"id":"9bfbb50a.54833","type":"mp4frag","z":"26b5a4eb.db1d34","uniqueName":"front_porch_main","hlsListSize":"4","contextAccess":"none","httpRoutes":true,"x":710,"y":20,"wires":[[]]},{"id":"387294df.f4c8f4","type":"ui_mp4_player","z":"26b5a4eb.db1d34","group":"ea204528.5e15d8","order":4,"width":"6","height":"4","name":"","sourceType":"url","sourceValue":"http://192.168.1.85:1880/front_porch_main.m3u8","aspectratio":"stretch","x":910,"y":60,"wires":[[]]},{"id":"ea204528.5e15d8","type":"ui_group","name":"Group 1","tab":"ef1a3eec.2694d","order":1,"disp":true,"width":6},{"id":"ef1a3eec.2694d","type":"ui_tab","z":"26b5a4eb.db1d34","name":"Dashboard","icon":"dashboard","order":1,"disabled":false,"hidden":false}]