How to build a video surveillance system from scratch?

General
1

Hi folks,

It is time for me to build a new video surveillance system from scratch. I have bought an usb SSD disc for my Raspberry Pi 4, and I would like to build this system based on the nodes from @kevinGodell (who had excellent testers like @SuperNinja, @krambriw, @wb666greene and many other folks from this community...).

But the discussions have become "very" long, and I am not sure which things are still relevant. I assume that a lot of issues have already been solved by Kevin, and that some solutions have meanwhile gotten successors with better performance/features. So I would appreciate VERY to get some thoughts on how to start, based on the best practices they have learned throughout these discussions.

Not sure if everything of my needs is currently already possible, or which node I should use for what:

  1. What is the best way to capture high resolution video from my IP cam. Is that via RTSP? And is audio easily included?

  2. When I want to do some processing on the individual images of that stream (e.g. face recognition), and optionally create a stream again from the updated images. What is the most performant way to do that?

  3. The video should be stored on my SSD disc. I assume I should use somehow small mp4 videos of N seconds length? And can I use mp4 fragments for this purpose?

  4. I want to be able to play the live video or recorded video on my Node-RED dashboard. What is the best way to play a lower resolution video? And do I need the upcoming version of the dashboard for the new socket.io version?

Thanks a lot !!
Bart

2 Likes
2

Hello Bart,
looks like you are never running out of ideas :sunglasses:

for the basic part of my surveillance system i'm using the below setup

  • Old 4GB 2Ghz laptop with Ubuntu docker and SSD drive
  • Node-Red is running in docker on this system
  • I use MotionEye for my basic video surveillance as well on this docker
  • the system can store video-files(high-res) and images per motion-event
  • I have setup 6 RTSP (cheap Chinese 4k cameras) cameras
  • 2 cameras are configured for motion detection (front of house)
  • remaining are just taking regular snapshots (every minute) of surrounding of house
  • all recordings are stored on my local NAS (accessible form Node-Red and even PLEX)
  • all motion alerts are triggered via webhooks to Node-Red which is sending the alert-video via telegram to my phone and showing the video on my dashboard
  • all camera live-streams are visible in Node-Red
  • I maintain an alert-list in Node-Red with the corrosponding Video-file-name (stored on NAS) and I'm able to replay any alert on request

The overall system is very performant (laptop is max 60% utilized) when doing motion-detection for 2 camera's. I did try with motion-detection with 4 camera's but than the utilization goes up to 90%.

I have done some trial-run with face and object detection and it was really good in regards of performance and quality during day but my cheap camera's quality is not always sufficient at night.

Code for live-Streaming cameras:

[{"id":"769295e6.93ea2c","type":"ui_template","z":"ab5f702e.d6756","group":"ba96c8f1.4fb378","name":"Just a view of cameras","order":4,"width":18,"height":12,"format":"<!DOCTYPE html>\n<html>\n<style>\nimg {\n    border: 1px solid #ddd;\n    border-radius: 8px;\n    padding: 2px;\n    width: 905px;\n}\n</style>\n\n<script type=\"text/javascript\">\nscope.$watch('msg', function(msg) {\n    inMessage(msg.payload);\n});\n\nfunction updatePic() {\n    document.getElementById('c51').src = \"http://192.168.2.202:8083\";\n    document.getElementById('c52').src = \"http://192.168.2.202:8084\";\n    document.getElementById('c53').src = \"\";\n    document.getElementById('c54').src = \"\";\n    document.getElementById('c55').src = \"\";\n    document.getElementById('c56').src = \"\";\n    document.getElementById('c57').src = \"\";\n    document.getElementById('c58').src = \"\";\n    //alert('updated pics');\n}\n\nfunction inMessage(event) {\n    if (event.match('checkWeb')) {\n        updatePic();\n    }\n}\n\n</script>\n<center>\n<table>\n    <tr><!-- Row 1 -->\n    \t\t<td style=\"text-align: center\"><img src=\"\" id=\"c51\" /></td>\n    </tr>\n    <tr><!-- Row 2 -->\n    \t\t<td style=\"text-align: center\"><img src=\"\" id=\"c52\" /></td>\n    </tr>\n</table>\n</center>\n</html>","storeOutMessages":false,"fwdInMessages":false,"resendOnRefresh":false,"templateScope":"local","x":620,"y":1600,"wires":[[]]},{"id":"bb5fdb34.e3dd88","type":"inject","z":"ab5f702e.d6756","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"600","crontab":"","once":true,"onceDelay":"10","topic":"","payload":"checkWeb","payloadType":"str","x":390,"y":1600,"wires":[["769295e6.93ea2c"]]},{"id":"ba96c8f1.4fb378","type":"ui_group","name":"Test","tab":"df07ccac.714e9","order":12,"disp":true,"width":"18","collapse":false},{"id":"df07ccac.714e9","type":"ui_tab","name":"LiveVideo","icon":"dashboard","disabled":false,"hidden":false}]

Code to stream a file from NAS

[{"id":"4eea8074.110a6","type":"ui_template","z":"fc48425a.4aeab","group":"2a107b87.e25264","name":"Vid_screen","order":2,"width":12,"height":7,"format":"<div>\n     <video id=\"video\" width=\"100%\" heigth=\"100%\"></video>\n</div>\n\n<script> \n    (function(scope) {\n        // Watch for messages being send to this template node\n        scope.$watch('msg', function (msg) {      \n            if (msg) {\n\n                var video = document.getElementById('video');\n                video.src = msg.filename;\n                video.play();\n            }       \n         }); \n    })(scope); \n</script>\n\n\n\n","storeOutMessages":false,"fwdInMessages":true,"resendOnRefresh":true,"templateScope":"local","x":1610,"y":320,"wires":[["68cfebac.08e884"]]},{"id":"2a107b87.e25264","type":"ui_group","z":"","name":"Last Event","tab":"1f273d9.25be8c2","order":1,"disp":true,"width":"12","collapse":false},{"id":"1f273d9.25be8c2","type":"ui_tab","z":"","name":"Security","icon":"fa-video-camera","order":2,"disabled":false,"hidden":false}]

I'm curious on how your solution is progressing ... good luck

2 Likes
3

ffmpeg. RTSP can carry audio with the video. The only problem that can occur is the type of audio codec in the ip cam may not be the right type of audio that can be played in the browser, which would require audio encoding and uses a bit of cpu, but not too much. Some of my newer cams have audio that is already encoded with the aac codec, which can easily be played in the browser. Maybe yours does?

The best way would be to use the 2nd video stream that should be available on your rtsp ip cam. That will be the least resource hog since you can just stream copy it without re-encoding. I always use both the main stream and sub streams from my cams so that I can stream the lower resolution version when in a slow wireless network.

You really don't need the socket. io player. You can just use hls.js to play the hls.m3u8 file. It is much more stable compared to my custom player.

As for the other questions , I don't have any good answers since I only do 24/7 recording without any motion/object detection. ffmpeg takes care of writing mp4 video all day long, broken into individual videos of 15 minutes duration. I have it written to a 6TB WD purple in an external enclosure. Once a day a script runs to unlink old videos more than 2 weeks old.

1 Like
4

Hey Bart, for object detection some guy did this node node-red-contrib-tfjs-coco-ssd (node) - Node-RED which seems to work quite well. Worth a look I think.

1 Like
5

If you're not already aware of it you might want to look at Frigate GitHub - blakeblackshear/frigate: NVR with realtime local object detection for IP cameras
Although it says 'designed for HomeAssistant' it actually runs completely independently in Docker and interfaces via mqtt. I have built a surveillance system in NodeRed using Frigate and mqtt with event snapshots sent to my phone from NodeRed by Telegram.

5 Likes
6

I concur!! Excellent tool. Not linked to Home Assistant if one doesn't want. The question was "from scratch". Using frigate might by too easy for @BartButenaers :wink:

1 Like
7

Hi all,
Thanks for all the feedback, to help me getting started with this.

I truly understand that people use MotionEye and other third-party tools, since those tools are good and you have in a rather short time good results.

But since the day I started using Node-RED, I wanted to have video surveillance fully integrated in Node-RED (without having to install and setup a third party tool). Of course it became quickly clear that ffmpeg is required (as third party library), because there is no alternative library available in javascript (which supports so much formats). Of course we can learn a lot from those third party tools, to avoid reinventing hot water.

Since people are very enthousiastic about Kevin's mp4 nodes, I would really like to try them. But to be honest, I haven't followed the related discussions in detail. And when I try to read them now, the information is kind of mindblowing for me...

So I would like to get a grip on the theory first, before I start building flows or nodes (for missing functionality). Would like to know the entire process flow that is being followed by every image that is being captured by an IP camera, until we see it on our dashboard.

The following diagram is FAR from correct or complete:

image
image947Ă—633 73.8 KB

But I hope this can be a starting point for a discussion.
You are all invited to copy my drawing into a paint program, and start drawing arrows, adding comments, and so on...
Would be nice to see file names, directories, missing arrows, content of output images, which data is fetched from to browser or pulled to the browser, and so on...

Hopefully at the end we have a detailed overview, that people can use as a guideline to setup a video surveillance system.

Thanks again !!!

1 Like
8

BTW in the mp4frag node config screen, you can see keyframe segements and playlist segments. If stuff is buffered in memory or on disk, it would be nice to have this visualised also in the flow.

And not sure if it is clearly visualized, but the 4 file icons (at the bottom of the drawing) represent mp4 files stored on disc: recorded footage to view later on.

9

Frigate looks good, I may need to learn about docker to package up my system.

From my experience over the past few years the idea of using some simple "motion detection" to crop out a region to be sent to the object detector, while logical, may in fact be counter productive. I run 7 4K cameras and 7 1080p cameras and I just resize the full image to the object detector size and run it. Counter-intuitively it works exceptionally well.

Here is a non-alert detection, rejected because the object being outside the borders of my property:

17_33_41.0_HummingbirdRight_Idle_AI
17_33_41.0_HummingbirdRight_Idle_AI1920Ă—1080 384 KB

Like Frigate, I use a Coral TPU and MobilenetSSD_v2_coco, the biggest limitation is decoding multiple rtsp streams.

A Jetson Nano with Coral TPU can process five 3 fps 4K cameras and hit an aggregate frame rate of 14.6 fps -- effectively running the AI on every frame.

My design goal was very different, I consider the 24/7 video recording problem solved by things like Zoneminder, Motion, Motioneye, etc, in the open source world and many reasonably priced commercial offerings, I wanted the AI to monitor the video streams and alert only when a person is detected on my property and work with an recording setup than can forward me the rtsp streams or image snapshots.

While I fully understand Bart's dedication to node-red and DIY, I wish he'd focus his magnificent talent on something other than re-inventing this particular wheel.

I do welcome a thread to clarify how to install and use Kevin's magnificent work, as making this easier to use opens up a lot of possibilities. I've been following it but not had time to actually play with it and its development has been so rapid and dynamic I've lost track of how to actually get started beyond a couple of Krambriw's sample flows and sub-flows that I've saved.

My final point is that monitoring live security camera feeds is incredibly boring so much so that I never look at them unless something has happened at which point the AI person detection images provide a great index into where the video might actually be interesting. A few days ago our mail carrier was attacked a block away from my house, luckily she escaped unharmed, but I reviewed my street facing camera video in hopes I could get a good high quality image of the fleeing suspect, no such luck, as apparently he went in the opposite direction, but I did see several police cars circling the neighborhood looking for the perp. I never knew any of this had happened until yesterday.

1 Like
10

Hi @wb666greene,
Well in fact I know that you are right... Damn you! But it is near the top of my bucket list, so I can't easily remove it :slight_smile:

In fact that was my goal of this discussion. But I might have said that a bit more explicit in the introduction...

11

The 1st thing we have to do is figure out how what type of video/audio streams are available. Whenever you get a chance to get to the command line, run ffprobe on your input. Be sure to use quotes around the input if it has special characters in it. Change the input url to whatever your cam's docs recommend. Also, try this on the sub stream, if there is one.

kevinGodell ~ $ ffprobe -hide_banner -i "rtsp://username:password@192.168.1.32:554/cam/realmonitor?channel=1&subtype=0"
Input #0, rtsp, from 'rtsp://admin:Purple@2026@192.168.1.32:554/cam/realmonitor?channel=1&subtype=0':
  Metadata:
    title           : Media Server
  Duration: N/A, start: 0.128000, bitrate: N/A
    Stream #0:0: Video: h264 (Main), yuv420p(progressive), 2688x1520, 100 tbr, 90k tbn, 180k tbc
    Stream #0:1: Audio: aac (LC), 8000 Hz, mono, fltp

edit: a better cmd to show more details:

ffprobe -hide_banner -i "rtsp://admin:Orange2021@192.168.1.32:554/cam/realmonitor?channel=1&subtype=0" -show_format -show_streams -print_format json
1 Like
12

That is indeed a good one!!! It are all rather old camera's, so I don't have high hopes. But I am going to buy new ones, as soon as I have a decent setup in Node-RED running.
But first I need to find some time to get my new RPI 4 running with a Samsung SSD disc that I bought last week. As soon as I managed to do that, I will get back here ...

13

Since you are still shopping, I will give a little recommendation of products that worked good for me so far.

I recently added an external drive to my pi 4 for a place to put my 24/7 recordings. It was tricky to find a good external enclosure that had its own separate power supply and could also be recognized after a power loss or reboot. After much tinkering and a firmware update, I was able to make this Fideco and 6TB WD Purple work for me. The original problem it had was that the stock firmware was so old that if I rebooted the pi while leaving the enclosure powered on, the usb communications would fail.

As for cameras, I have been playing with Amcrest lately. I have moved on from using the cheapest chinese no-brand cams to something that costs a little bit more, around $60 USD. I was pleasantly surprised at the quality and performance. They actually have a web interface that does not require internet explorer or flash player for viewing. They can also handle multiple connections. The cams with audio are already encoded with aac, which makes stream copying them much easier and use less cpu load in ffmpeg. cams tested: 5mp poe no audio, 5mp poe with audio

1 Like
14

I concur about Amcrest IP cams, another good source for Cameras is "Urban Security Group" ( https://www.cctv.supplies/ ). They sell direct or on Amazon (the selection is larger direct) but they have a 90 day no questions asked money back guarantee which I can verify that they honor as they refunded my money on a rather expensive 4K 5-50X zoom camera I'd got for a plate reader project, it worked great in daylight but was unusable at night and when a firmware update they sent me didn't fix it, they refunded my money without hassles.

15

A post was split to a new topic: Problem with video feed

16

Hi mister @kevinGodell,
I only had 2 hours available yesterday to play with your nodes, but I'm very impressed ...
Nice work!!

1 Like
17

Hey Kevin,
I have one Hikvision cam, so going to test that one now.
Your ffprobe command gives me following Video information (not Audio):

  Metadata:
    title           : Media Presentation
  Duration: N/A, start: 0.300000, bitrate: N/A
    Stream #0:0: Video: h264 (Main), yuvj420p(pc, bt709, progressive), 2688x1520 [SAR 1:1 DAR 168:95], 20 fps, 20 tbr, 90k tbn, 40 tbc

When I login the web interface of the camera then I see these settings (i.e. no AAC audio):

image
image936Ă—429 20.1 KB

When I change the stream's video type to both video and audio:

image

Then indeed your probe command gives me both video and audio information:

  Metadata:
    title           : Media Presentation
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: h264 (Main), yuvj420p(pc, bt709, progressive), 2688x1520 [SAR 1:1 DAR 168:95], 20 fps, 20 tbr, 90k tbn, 40 tbc
    Stream #0:1: Audio: pcm_mulaw, 8000 Hz, 1 channels, s16, 64 kb/s

Do you have any recommendations for the audio/video settings?
And can you please guide me a bit of how to compose a good ffmpeg command, that is able to decode the rtsp stream?

18

In the camera's gui, I would set the i frame interval to the lowest settings that it allows. Since we will be stream copying, that setting will have a direct affect on the segment duration. Shorter segments is usually better for live streaming. For example, if your fps is set to 20 and the i frame is set to 40, then you will have segments with a duration approximately 2 seconds long.

Another thing to consider is the byte size of the segments. Having a higher bitrate equals bigger segments, which is then slower over a network. Lower the quality settings to the least acceptable. That is something you will have to tweak many times to get right.

The profile is something that I never really noticed making much of a difference. Higher profiles add extra data, which allows more compression, but then becomes less compatible with older devices.

Your audio cannot be stream copied and played in the browser since there is no AAC option. It will have to be encoded if you want audio with the video in the browser, which might cost about 5% additional cpu load on the ffmpeg instance.

We should start off with the simplest ffmpeg command and only add extras as needed. If you are going to pipe the content into node-red-contrib-mp4frag, then you can use something like this:

edit:
ffmpeg -f rtsp -rtsp_transport tcp -i rtsp://your.cameras.url -f mp4 -c:v copy -c:a aac -movflags +frag_keyframe+empty_moov+default_base_moof pipe:1

Let me know if that gets working, and then we can get a little more advanced.

2 Likes
19

Splendid Noob level info. Just what I needed, to get started step by step, and gain understanding...
Will get back to you later on. It is my little boy's birthday today. And he has planned a lot of activities :wink:

2 Likes
20

As you can see in the screenshot above about my camera configuration, I have 20 fps and an I-frame interval of 50. So I expect every segment to contain 2,5 seconds of video footage? But when I use a node-red-contrib-msg-speed node, then I see that have on average 4 messages per 5 seconds:

image

Based on your explanation I would have expected only two messages to arrive every 5 seconds...
Do you have an idea what I am doing wrong?

Is there an easy way to display those segments e.g. using a node-red-contrib-image-output node (as jpeg images...)? Just to make sure that my segments contain uncorrupted images, and to be able to determine whether the quality is good (e.g. while experimenting to find an acceptable bitrate).