[ANNOUNCE] node-red-contrib-multipart-stream-decoder : version 1.0.0 beta

Share Your Nodes
1

Hi folks,

Long time between my announcements. But let's be honest: it would be a lousy Christmas without presents under the Node-RED community tree...

The last version of my node-red-contrib-multipart-stream-decoder node was already more than 4 years old. Meanwhile I had quite a lot of questions to implement digest authentication (a.o. from @dynamicdave), because nowadays most IP camera's require it when you want to read an MJPEG stream.

Since my new Amcrest camera's also use digest authentication, I had no other choice than to implement it. And here it is finally in version 1.0.0 :champagne: :clinking_glasses: :partying_face:

image

Must admit that it was not easy to get it done, because it seems that only the "request" lib offered pause and resume functionality. But the other libraries ("got", "axios", "node-fetch") not... Tried them all, but at the end I have sticked to axios, and I workaround it.

Short demo to get in to the christmas mood :christmas_tree: :santa: :gift:
Although I have to admit that at this moment of the evening, it is pretty dark and there is not much movement in the Christmas markets as you can see :wink:

demo_mjpeg

[{"id":"50a4d182a71cdf80","type":"inject","z":"fbee74db83781e91","name":"Stop stream","props":[{"p":"stop","v":"true","vt":"bool"}],"repeat":"","crontab":"","once":false,"onceDelay":"","topic":"","x":530,"y":3240,"wires":[["8c35709.24b749"]]},{"id":"731c40362af5e27a","type":"inject","z":"fbee74db83781e91","name":"Pause stream","props":[{"p":"pause","v":"true","vt":"bool"}],"repeat":"","crontab":"","once":false,"onceDelay":"","topic":"","x":530,"y":3140,"wires":[["8c35709.24b749"]]},{"id":"382a1962b0da9b0d","type":"inject","z":"fbee74db83781e91","name":"Resume stream","props":[{"p":"resume","v":"true","vt":"bool"}],"repeat":"","crontab":"","once":false,"onceDelay":"","topic":"","x":520,"y":3200,"wires":[["8c35709.24b749"]]},{"id":"ee86d795.1e8cb8","type":"inject","z":"fbee74db83781e91","name":"First stream (Czech Republic)","props":[{"p":"url","v":"http://89.203.137.209/mjpg/video.mjpg","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":"","topic":"","x":440,"y":3040,"wires":[["8c35709.24b749"]]},{"id":"8c35709.24b749","type":"multipart-decoder","z":"fbee74db83781e91","name":"","ret":"bin","url":"","tls":"","authentication":"digest","delay":0,"maximum":"1000000","blockSize":"1","enableLog":"on","credentials":{},"x":750,"y":3040,"wires":[["a47f8c9bfe0c0793"],[]]},{"id":"a47f8c9bfe0c0793","type":"image","z":"fbee74db83781e91","name":"","width":"400","data":"payload","dataType":"msg","thumbnail":false,"active":true,"pass":false,"outputs":0,"x":980,"y":3040,"wires":[]},{"id":"c6cf7a07de508770","type":"inject","z":"fbee74db83781e91","name":"Second stream (Marktplatz Austria)","props":[{"p":"url","v":"http://cam1.rauris.net/axis-cgi/mjpg/video.cgi","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":460,"y":3080,"wires":[["8c35709.24b749"]]}]

P.S. At the moment of this announcement, it seems the streams get mixed up when you switch too fast between different streams. I didn't analyze that yet...

As always, it would be nice if some folks could test it. Since it is not on NPM yet, you can install it from my Github account using following command (from within your .node-red folder):

npm install bartbutenaers/node-red-contrib-multipart-stream-decoder

Enjoy it!!!
Bart

9 Likes
2

In the version of Github I have now used an AbortController, to interrupt the async background loop that is reading chunks from the active response. By doing it this way, the active response is immediately being stopped. That way it seems to work much better when switching streams very fast.

2 Likes
3

@BartButenaers, though I don't have an immediate need for this node I do want to thank you again for yet another excellent piece of work. Your contributions are always among the best, if not THE best, ever!

4

@LosinIt,
Thanks for the kind response! That is the fuel that keeps us going...
Got lots of unfinished nodes, and lots of ideas for new nodes. But very few free time...

Will explain my use case: was working with our friend @dceejay on an object detection node, but needed a way to capture a few images per seconds from my new camera's for this (with low CPU usage on my RPI4). So I had setup an mjpeg stream on my cams, with very low frame rate (3 images per second), which I wanted to send to my Google Coral TPU USB stick. But my new cams required digest authentication for their MJPEG streams...

Now back to the object detection, because it contains a nasty memomry leak. Which blows up the RAM of my RPI4 in less than a minute, which means the object detection is not possible at the moment...

2 Likes
5

Nice Bart!!
In my case I cannot stop the stream from Austria, it just continues even if I disable the node!!

Best regards, Walter

EDIT: Sorry, works fine now after a second node-red-restart

image
image1253Ă—732 223 KB

6

Hi Walter,

No problem. Good to hear that it works, because I had no explanation for that.

BTW I have added a new option to the config screen in this version:
image

Which was a tremendous help for me during development, in order to find what was going on behind the scenes (in the async processing):

image
image798Ă—115 4.64 KB

1 Like
7

Ah and this might also be of interest to some of you. There are a lot of websites containing links to public camera's, but those links are often completely dead.

So I figured out my own way to find (mjpeg or m3u8) streams of public camera's to test:

8

A bit off topic, but which Amcrest cameras did you go for?

9

Hi Bart

I had the same problem as krambriw, with the same solution (restart Node-red a second time)

The stream from Poland seems to have gone AWOL

I do not have a 'debug log' dropdown in the config screen of the version I just obtained with npm install bartbutenaers/node-red-contrib-multipart-stream-decoder

10

Completely no idea what could cause that. How weird is that...

Did you install it from within your .node-red folder, and did you refresh your browser window?

11

I did. This is the first time I installed it so there is no previous version

12

That I also don't understand. The install command above contains "bartbutenaers", which means you should get it from my Github repo. And that debug log field is in my repo. Without conditions, so it should always be visible...

13

If I reinstall it can I just re-run that.

I had the jitter between 2 streams happen again and I pressed stop but nothing happened (still streaming), yet when I then pressed pause I got the message

Useless to send msg.pauze when no stream is active

14

If you would be able to figure out at which conditions it happens, that would be very helpful.
Once I know the steps how to reproduce that problem, then I can start digging to find the root cause.

15

I had a stream running, then without pressing stop streaming I selected other streams.

This work fine until I selected a stream that wouldn't connect

The first click didn't do anything, second gave a "Error: socket hang up"., third gave a 'Requesting' status comment until an Econnreset status message

Clicking back onto the first stream was OK, but then selecting another working stream it all went wonky.

Flashing between the two working streams, stop stream not working.

So, to summarize it would seem that it is the selection of a stream that doesn't connect is the issue

Only a node-red restart sorts it out

16

@Buckskin,
Thanks a lot for your analysis! Really appreciate if somebody takes the time to test a beta version...
Will try find some time tonight to analyze this...

17

@Buckskin,
I have created following flow, containing a failing stream and two working streams:

[{"id": "a7a459ea5976ad90","type": "inject","z": "8b52e098cd5f73fd","name": "Stop stream","props": [{"p": "stop","v": "true","vt": "bool"}],"repeat": "","crontab": "","once": false,"onceDelay": "","topic": "","x": 450,"y": 3500,"wires": [["6016ba8d1ed71da2"]]},{"id": "b21c56676839da39","type": "inject","z": "8b52e098cd5f73fd","name": "Failing stream","props": [{"p": "url","v": "http://89.203.137.209/mjpg/video.mjpg","vt": "str"}],"repeat": "","crontab": "","once": false,"onceDelay": "","topic": "","x": 450,"y": 3380,"wires": [["6016ba8d1ed71da2"]]},{"id": "6016ba8d1ed71da2","type": "multipart-decoder","z": "8b52e098cd5f73fd","name": "","ret": "bin","url": "","tls": "","authentication": "none","delay": 0,"maximum": "1700000","blockSize": "1","enableLog": "on","x": 670,"y": 3380,"wires": [["d192c5804936bc74"],[]]},{"id": "d192c5804936bc74","type": "image","z": "8b52e098cd5f73fd","name": "","width": "640","data": "payload","dataType": "msg","svg": "svg","svgType": "msg","thumbnail": false,"active": true,"pass": false,"outputs": 0,"x": 900,"y": 3380,"wires": []},{"id": "5a70a37216a61b90","type": "inject","z": "8b52e098cd5f73fd","name": "Working stream 1","props": [{"p": "url","v": "http://cam1.rauris.net/axis-cgi/mjpg/video.cgi","vt": "str"}],"repeat": "","crontab": "","once": false,"onceDelay": 0.1,"topic": "","x": 440,"y": 3420,"wires": [["6016ba8d1ed71da2"]]},{"id": "9ee23df09b57ecb5","type": "inject","z": "8b52e098cd5f73fd","name": "Working stream 2","props": [{"p": "url","v": "http://79.141.146.81/mjpg/video.mjpg","vt": "str"}],"repeat": "","crontab": "","once": false,"onceDelay": 0.1,"topic": "","x": 440,"y": 3460,"wires": [["6016ba8d1ed71da2"]]}]

Have tried all kind of combinations to switch between the streams, but it doesn't go crazy whatever I try. As you can see in this (gifcam) animation:

cannot_reproduce

18

I managed it. When it DID go wonky both images went. I am going to try and reinstall the node to see if that makes any difference

Edit: I now have version 1.0, with the debug dropdown so I will try it again

19

This second flow also works fine & as expected in my installation. I only had the issue first time immediately after the installation of the new node version, after a second node-red-restart it never appeared again

  1. had the previous version installed
  2. installed the new version in /home/pi/.node-red
  3. node-red-restart
  4. issue was detected
  5. node-red-restart
  6. issue no more

So it seems "something" was there or became "repaired"

I think best way to verify would be to install in another machine/node red install

20

Installed it in another Pi where there was no previous version. Here both flows worked perfectly well after a singe node-red-restart. No issues found!

image
image1253Ă—732 238 KB

1 Like