Object Detection using node-red-contrib-tfjs-coco-ssd

SuperNinja · 10 March 2020 14:41

Hi all ,
i try to use @dceejay node-red-contrib-tfjs-coco-ssd to detect object in picture with the example attached :

[{"id":"a2e68203.7be26","type":"tensorflowCoco","z":"d0cece46.87ac6","name":"","model":"","x":510,"y":1160,"wires":[["f7c2654b.8582b8","92bee663.a421b8","19ca5cc7.3b49d3"]]},{"id":"f7c2654b.8582b8","type":"debug","z":"d0cece46.87ac6","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":810,"y":1200,"wires":[]},{"id":"92bee663.a421b8","type":"debug","z":"d0cece46.87ac6","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"classes","targetType":"msg","x":810,"y":1240,"wires":[]},{"id":"69f3ad28.a4b994","type":"base64","z":"d0cece46.87ac6","name":"","action":"str","property":"payload","x":500,"y":1100,"wires":[["a4ad37b6.b74438"]]},{"id":"a4ad37b6.b74438","type":"template","z":"d0cece46.87ac6","name":"","field":"payload","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"<img width=\"320px\" height=\"240px\" src=\"data:image/jpg;base64,{{{payload}}}\">","output":"str","x":640,"y":1100,"wires":[["b54432f0.8059a"]]},{"id":"b54432f0.8059a","type":"ui_template","z":"d0cece46.87ac6","group":"aef761dd.7e9dd","name":"","order":2,"width":7,"height":5,"format":"<div ng-bind-html=\"msg.payload\"></div>","storeOutMessages":true,"fwdInMessages":true,"templateScope":"local","x":800,"y":1100,"wires":[[]]},{"id":"19ca5cc7.3b49d3","type":"ui_table","z":"d0cece46.87ac6","group":"aef761dd.7e9dd","name":"","order":15,"width":9,"height":3,"columns":[],"outputs":0,"cts":false,"x":790,"y":1160,"wires":[]},{"id":"dd9cd04b.d370a","type":"http request","z":"d0cece46.87ac6","name":"","method":"GET","ret":"bin","paytoqs":false,"url":"https://loremflickr.com/320/240/sport","tls":"","persist":false,"proxy":"","authType":"","x":330,"y":1100,"wires":[["a2e68203.7be26","69f3ad28.a4b994"]]},{"id":"9c31f01c.dd7c4","type":"ui_button","z":"d0cece46.87ac6","name":"","group":"aef761dd.7e9dd","order":13,"width":7,"height":1,"passthru":false,"label":"New Picture","tooltip":"","color":"","bgcolor":"","icon":"","payload":"","payloadType":"str","topic":"","x":170,"y":1100,"wires":[["dd9cd04b.d370a"]]},{"id":"aef761dd.7e9dd","type":"ui_group","z":"","name":"Object detection","tab":"3dce94d.cb4316c","disp":true,"width":"9","collapse":false},{"id":"3dce94d.cb4316c","type":"ui_tab","z":"","name":"Home","icon":"dashboard"}]

The readme is like this :

A Node-RED node for Object Detection using TensorFlowJS CoCo SSD.

NOTE : Tensorflow.js is only available on certain OS/Hardware/processor combinations. It does not autimatically install/work on all platforms. You may be unlucky.

The installation with the palette went well with the addition of the ui-table node. But I have this error:
Error: Cannot find module 'C:\Users\christian\.Node-RED-Desktop\node_modules\@tensorflow\tfjs-node\lib\napi-v4\tfjs_binding.node'Require stack:- C:\Users\christian\.Node-RED-Desktop\node_modules\@tensorflow\tfjs-node\dist\index.js- C:\Users\christian\.Node-RED-Desktop\node_modules\node-red-contrib-tfjs-coco-ssd\tfjs.js- C:\Users\christian\AppData\Local\Programs\node-red-desktop\resources\app.asar\node_modules\@node-red\registry\lib\loader.js- C:\Users\christian\AppData\Local\Programs\node-red-desktop\resources\app.asar\node_modules\@node-red\registry\lib\index.js- C:\Users\christian\AppData\Local\Programs\node-red-desktop\resources\app.asar\node_modules\@node-red\runtime\lib\nodes\index.js- C:\Users\christian\AppData\Local\Programs\node-red-desktop\resources\app.asar\node_modules\@node-red\runtime\lib\index.js- C:\Users\christian\AppData\Local\Programs\node-red-desktop\resources\app.asar\main\node-red.js- C:\Users\christian\AppData\Local\Programs\node-red-desktop\resources\app.asar\main\config-manager.js- C:\Users\christian\AppData\Local\Programs\node-red-desktop\resources\app.asar\main\main.js-

Is something missing from the installation, or am I just "out of luck"?
Should I install node-red-contrib-tfjs? I can't find it in the manage palette.

P.S. I use Node-Red Desktop on Win10 64Bits (NR version v1.0.3, Node.js v12.8.1)

[EDIT] Compatible material

To date, here is the list of devices compatible and not, with this node (this list will be updated according to future testers) :

Raspberry PI 3B and 3B+
Mac
Lenovo with Debian
WIN10 PC with nodejs : v12.16.1 & npm : 6.13.4

Unsupported hardware (at the time of this writing)

Pi Zero W
NVIDIA jetson Nano

krambriw · 10 March 2020 17:38

Nice!!!
I did just install the node (and the ui_table that I didn't have installed already)

krambriw · 11 March 2020 05:13

Very impressive I must say. On a Pi3 it takes just around 1000 ms for the node to analyze the picture for objects. That is by far the best I have seen on a Pi!!!

The node also supports pictures delivered via mqtt. Next, I have to try this on one of my Nvidia Jetson Nano where I have the GPU waiting for this
EDIT: This failed, same error as Christian (@SuperNinja) reported above, seems not to be compatible with the Nvidia platform

Challenge, would be nice to have a rectangle overlayed in the picture representing the box coordinates

SuperNinja · 11 March 2020 06:58

This is what i'm working on ! Screenshot_20200311-075403|690x388, 50%

krambriw · 11 March 2020 07:15

Very nice!!!

Myself, trying to figure out how to make this work with a GPU. I have the Nvidia stuff and expect it should increase performance rather significant (if I will figure out).

Tried it already also with an old laptop running debian, analyze is ready in approx 200 ms

SuperNinja · 11 March 2020 12:16

Challenge : rectangle overlayed in the picture : NEED HELP !
In this test i display only the first class

As you can see the "person" is detected, the box displayed BUT the picture flash 1ms and disapear !

[{"id":"24bd5862.38b1b8","type":"ui_template","z":"a8b1b208.7036f","group":"7c26c8d0.c1e658","name":"highlightPict","order":2,"width":9,"height":6,"format":"<!DOCTYPE html>\n<html>\n<style type=\"text/css\"> \n.wrapper {\n    position:relative;\n    width:320px;\n    height:240px;\n    border:2px solid grey;\n}\n\n.box {\n    position:absolute;\n    top:{{msg.y}}px;\n    left:{{msg.x}}px;\n    width:{{msg.width}}px;\n    height:{{msg.height}}px;\n    border:2px solid red;\n    background-color:transparent\n}\n </style> \n<body>\n\n\n<div class=\"wrapper\">\n     <img src=\"data:image/jpg;base64,{{msg.payload}}\"  />\n     <div class=\"box\"></div>\n</div>\n</body>\n</html>\n\n\n\n<!--\n//////////////////////////////////////////////////////////////////////////////\n<!DOCTYPE html>\n<html>\n<head>\n<style> \n#example1 {\n  background-image: url(data:image/jpg;base64,{{msg.payload}});\n  border: 1px solid blue;  \n  background-size:auto 100% ;\n  background-repeat: no-repeat;\n  /*\n}\nhttp://newscrane.com/wp-content/uploads/2019/09/Car-insurance-Newscrane-02.jpg\nsvg {\n border: 1px solid green;\n}\n*/\n</style>\n</head>\n<body>\n<div id=\"example1\">\n<svg width=\"320\" height=\"240\" style=\"border: 1px solid lightgrey;\">\n  <rect x={{msg.x}} y={{msg.y}} width={{msg.width}} height={{msg.height}} style=\"stroke:red;stroke-width:2;fill-opacity:0\" />\n  Sorry, your browser does not support inline SVG.  \n</svg>\n</div>\n</body>\n</html>\n-->\n<!--\n<img width=\"100%\" height=\"100%\" alt=\"HighLight Picture\" src=\"data:image/png;base64,{{msg.payload}}\" />\n<span style=\"font-size:10px;\">{{msg.fileName}} {{msg.affSlideIndex}}/{{msg.numSlides}}</span> \n-->\n","storeOutMessages":true,"fwdInMessages":true,"templateScope":"local","x":1120,"y":1800,"wires":[[]]},{"id":"86f3292b.6471b8","type":"tensorflowCoco","z":"a8b1b208.7036f","name":"","model":"","scoreThreshold":"","x":590,"y":1840,"wires":[["2665fee8.ccc712","5b1ba3a2.77ebdc","61d7c39d.c5efdc","acbb43ee.e63df","c2a407c5.827fb8"]]},{"id":"2665fee8.ccc712","type":"debug","z":"a8b1b208.7036f","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":770,"y":1900,"wires":[]},{"id":"5b1ba3a2.77ebdc","type":"debug","z":"a8b1b208.7036f","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"classes","targetType":"msg","x":770,"y":1870,"wires":[]},{"id":"f2874676.b4f3b8","type":"base64","z":"a8b1b208.7036f","name":"","action":"str","property":"payload","x":580,"y":1770,"wires":[["d215245.d4919d8","24bd5862.38b1b8"]]},{"id":"61d7c39d.c5efdc","type":"ui_table","z":"a8b1b208.7036f","group":"7c26c8d0.c1e658","name":"table detection","order":8,"width":10,"height":2,"columns":[],"outputs":0,"cts":false,"x":780,"y":1810,"wires":[]},{"id":"bf528beb.36b538","type":"http request","z":"a8b1b208.7036f","name":"","method":"GET","ret":"bin","paytoqs":false,"url":"https://loremflickr.com/320/240/sport","tls":"","persist":false,"proxy":"","authType":"","x":370,"y":1850,"wires":[["86f3292b.6471b8","f2874676.b4f3b8","97a97ff2.45acd","64c32917.ff8b28"]]},{"id":"fd369cc7.f262b","type":"ui_button","z":"a8b1b208.7036f","name":"","group":"7c26c8d0.c1e658","order":6,"width":1,"height":1,"passthru":true,"label":"New Picture","tooltip":"","color":"","bgcolor":"","icon":"","payload":"","payloadType":"str","topic":"","x":220,"y":1850,"wires":[["bf528beb.36b538"]]},{"id":"64c32917.ff8b28","type":"image-info","z":"a8b1b208.7036f","name":"","x":590,"y":1910,"wires":[[]]},{"id":"27cb047f.a8173c","type":"function","z":"a8b1b208.7036f","name":"posiRectangle","func":"//let pictureB64[] = msg.payload;\nlet box = msg.detect[0];\nmsg.x = box.bbox[0];\nmsg.y = box.bbox[1];\nmsg.width = box.bbox[2];\nmsg.height= box.bbox[3];\nmsg.class = box[\"class\"];\nmsg.score = box[\"score\"];\n\nreturn msg;\n\n\n\n\n\n/*\n252x200 : person devant chalet\n0: object\n    bbox: array[4]\n        0: 127\n        1: 86\n        2: 29\n        3: 90\n    class: \"person\"\n    score: 0.9015087080001831\n    \n    x=127 y=86 width=127-90  height=86-29\n*/","outputs":1,"noerr":0,"x":930,"y":1840,"wires":[["91b9bccd.feaa9","24bd5862.38b1b8"]]},{"id":"91b9bccd.feaa9","type":"debug","z":"a8b1b208.7036f","name":"rectangle","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":1110,"y":1840,"wires":[]},{"id":"97a97ff2.45acd","type":"image","z":"a8b1b208.7036f","name":"","width":"400","data":"payload","dataType":"msg","thumbnail":false,"active":true,"x":600,"y":1980,"wires":[]},{"id":"5f00cc2e.197f64","type":"inject","z":"a8b1b208.7036f","name":"test","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":80,"y":1850,"wires":[["fd369cc7.f262b"]]},{"id":"c2a407c5.827fb8","type":"change","z":"a8b1b208.7036f","name":"pay to detect","rules":[{"t":"move","p":"payload","pt":"msg","to":"detect","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":770,"y":1840,"wires":[["27cb047f.a8173c"]]},{"id":"d215245.d4919d8","type":"debug","z":"a8b1b208.7036f","name":"b64","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":890,"y":1720,"wires":[]},{"id":"7c26c8d0.c1e658","type":"ui_group","z":"","name":"detection","tab":"b123e6c8.d3e948","order":2,"disp":true,"width":"10","collapse":true},{"id":"b123e6c8.d3e948","type":"ui_tab","z":"","name":"Slide","icon":"dashboard","order":2,"disabled":false,"hidden":false}]

In an other hand , there is "nothing" to detect :

Detect nothing = the picture is loaded

This is the template node :

<!DOCTYPE html>
<html>
<style type="text/css"> 
.wrapper {
    position:relative;
    width:320px;
    height:240px;
    border:2px solid grey;
}

.box {
    position:absolute;
    top:{{msg.y}}px;
    left:{{msg.x}}px;
    width:{{msg.width}}px;
    height:{{msg.height}}px;
    border:2px solid red;
    background-color:transparent
}
 </style> 
<body>


<div class="wrapper">
     <img src="data:image/jpg;base64,{{msg.payload}}"  />
     <div class="box"></div>
</div>
</body>
</html>

I would like the image to persist, before displaying the box! How to do ?

krambriw · 11 March 2020 15:24

The problem here is that the image is overwritten by the msg from the posiRectangle. Somehow the image needs to be "saved" in the ui template and then used in the html code when the box data comes in

I tested a bit by instead saving the image to file and then load it in the html when the box data arrives and then it works but this is of course not a good way doing it

SuperNinja · 11 March 2020 15:30

not bad !!
that's for sure, it saves an image on the SD card, it's not good for its life.
We are not far from the goal, it would be nice to be able to combine the image + the box in a file and then display it in the HTML code
Is it possible to put in a function, the msg.payload of the image + msg.x, msg.y, msg.width ... so as to send everything at once? It will avoid overwriting the image on arrival of msg.x, y, ...

SuperNinja · 11 March 2020 16:04

It's alive !!!

[{"id":"86f3292b.6471b8","type":"tensorflowCoco","z":"a8b1b208.7036f","name":"","model":"","scoreThreshold":"","x":590,"y":1840,"wires":[["2665fee8.ccc712","5b1ba3a2.77ebdc","61d7c39d.c5efdc","c2a407c5.827fb8"]]},{"id":"2665fee8.ccc712","type":"debug","z":"a8b1b208.7036f","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":770,"y":1900,"wires":[]},{"id":"5b1ba3a2.77ebdc","type":"debug","z":"a8b1b208.7036f","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"classes","targetType":"msg","x":770,"y":1870,"wires":[]},{"id":"f2874676.b4f3b8","type":"base64","z":"a8b1b208.7036f","name":"","action":"str","property":"payload","x":580,"y":1770,"wires":[["d215245.d4919d8","cc75d53b.4b47b8"]]},{"id":"61d7c39d.c5efdc","type":"ui_table","z":"a8b1b208.7036f","group":"7c26c8d0.c1e658","name":"table detection","order":8,"width":10,"height":2,"columns":[],"outputs":0,"cts":false,"x":780,"y":1810,"wires":[]},{"id":"bf528beb.36b538","type":"http request","z":"a8b1b208.7036f","name":"","method":"GET","ret":"bin","paytoqs":false,"url":"https://loremflickr.com/320/240/sport","tls":"","persist":false,"proxy":"","authType":"","x":370,"y":1850,"wires":[["fb2a6f1a.6c4ed"]]},{"id":"fd369cc7.f262b","type":"ui_button","z":"a8b1b208.7036f","name":"","group":"7c26c8d0.c1e658","order":6,"width":1,"height":1,"passthru":true,"label":"New Picture","tooltip":"","color":"","bgcolor":"","icon":"","payload":"","payloadType":"str","topic":"","x":220,"y":1850,"wires":[["bf528beb.36b538"]]},{"id":"64c32917.ff8b28","type":"image-info","z":"a8b1b208.7036f","name":"","x":590,"y":1910,"wires":[[]]},{"id":"27cb047f.a8173c","type":"function","z":"a8b1b208.7036f","name":"posiRectangle","func":"var pictureBuffer = flow.get('pictureBuffer')|| [];\nmsg.pictureBuffer = pictureBuffer;\n//let pictureB64[] = msg.payload;\nlet box = msg.detect[0];\nmsg.x = box.bbox[0];\nmsg.y = box.bbox[1];\nmsg.width = box.bbox[2];\nmsg.height= box.bbox[3];\nmsg.class = box[\"class\"];\nmsg.score = box[\"score\"];\n\nreturn msg;\n\n\n\n\n\n/*\n252x200 : person devant chalet\n0: object\n    bbox: array[4]\n        0: 127\n        1: 86\n        2: 29\n        3: 90\n    class: \"person\"\n    score: 0.9015087080001831\n    \n    x=127 y=86 width=127-90  height=86-29\n*/","outputs":1,"noerr":0,"x":950,"y":1840,"wires":[["91b9bccd.feaa9","4d7ff98d.d0a008"]]},{"id":"91b9bccd.feaa9","type":"debug","z":"a8b1b208.7036f","name":"rectangle","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":1110,"y":1840,"wires":[]},{"id":"97a97ff2.45acd","type":"image","z":"a8b1b208.7036f","name":"","width":"400","data":"payload","dataType":"msg","thumbnail":false,"active":true,"x":600,"y":1980,"wires":[]},{"id":"5f00cc2e.197f64","type":"inject","z":"a8b1b208.7036f","name":"test","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":80,"y":1850,"wires":[["fd369cc7.f262b"]]},{"id":"c2a407c5.827fb8","type":"change","z":"a8b1b208.7036f","name":"pay to detect","rules":[{"t":"move","p":"payload","pt":"msg","to":"detect","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":770,"y":1840,"wires":[["27cb047f.a8173c"]]},{"id":"d215245.d4919d8","type":"debug","z":"a8b1b208.7036f","name":"b64","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":890,"y":1720,"wires":[]},{"id":"66276b4a.5c6cb4","type":"link in","z":"a8b1b208.7036f","name":"coco","links":["fb2a6f1a.6c4ed","39e7bc7c.c0a794","308e49b6.df1dc6"],"x":455,"y":1940,"wires":[["f2874676.b4f3b8","97a97ff2.45acd","64c32917.ff8b28","86f3292b.6471b8"]]},{"id":"fb2a6f1a.6c4ed","type":"link out","z":"a8b1b208.7036f","name":"","links":["66276b4a.5c6cb4"],"x":320,"y":1940,"wires":[]},{"id":"4d7ff98d.d0a008","type":"ui_template","z":"a8b1b208.7036f","group":"7c26c8d0.c1e658","name":"highlightPict","order":2,"width":9,"height":6,"format":"<!DOCTYPE html>\n<html>\n<style type=\"text/css\"> \n.wrapper {\n    position:relative;\n    width:320px;\n    height:240px;\n    border:2px solid grey;\n}\n\n.box {\n    position:absolute;\n    top:{{msg.y}}px;\n    left:{{msg.x}}px;\n    width:{{msg.width}}px;\n    height:{{msg.height}}px;\n    border:2px solid red;\n    background-color:transparent\n}\n\n.text {\n    position:absolute;\n    top:{{msg.y}}px;\n    left: {{msg.x}}px;\n    color: red;\n\n}\n </style> \n<body>\n\n\n<div class=\"wrapper\">\n    <img src=\"data:image/jpg;base64,{{msg.pictureBuffer}}\"/> \n     <div class=\"box\"></div>\n     <div class=\"text\">{{msg.class}}</div>\n     \n</div>\n</body>\n</html>\n <!-- \n\n-->\n<!--\n//////////////////////////////////////////////////////////////////////////////\n<!DOCTYPE html>\n<html>\n<head>\n<style> \n#example1 {\n  background-image: url(data:image/jpg;base64,{{msg.payload}});\n  border: 1px solid blue;  \n  background-size:auto 100% ;\n  background-repeat: no-repeat;\n  /*\n}\nhttp://newscrane.com/wp-content/uploads/2019/09/Car-insurance-Newscrane-02.jpg\nsvg {\n border: 1px solid green;\n}\n*/\n</style>\n</head>\n<body>\n<div id=\"example1\">\n<svg width=\"320\" height=\"240\" style=\"border: 1px solid lightgrey;\">\n  <rect x={{msg.x}} y={{msg.y}} width={{msg.width}} height={{msg.height}} style=\"stroke:red;stroke-width:2;fill-opacity:0\" />\n  Sorry, your browser does not support inline SVG.  \n</svg>\n</div>\n</body>\n</html>\n-->\n<!--\n<img width=\"100%\" height=\"100%\" alt=\"HighLight Picture\" src=\"data:image/png;base64,{{msg.payload}}\" />\n<span style=\"font-size:10px;\">{{msg.fileName}} {{msg.affSlideIndex}}/{{msg.numSlides}}</span> \n-->\n","storeOutMessages":true,"fwdInMessages":true,"templateScope":"local","x":1120,"y":1760,"wires":[[]]},{"id":"cc75d53b.4b47b8","type":"function","z":"a8b1b208.7036f","name":"flow.set pictureBuffer","func":"// Créé une variable pour stocker le pictureBuffer si inexistante\nvar pictureBuffer = flow.get('pictureBuffer')|| [];\npictureBuffer = msg.payload;\n// store the value \nflow.set('pictureBuffer',pictureBuffer);\n\n\n\n\n\n","outputs":0,"noerr":0,"x":680,"y":1690,"wires":[]},{"id":"7c26c8d0.c1e658","type":"ui_group","z":"","name":"detection","tab":"b123e6c8.d3e948","order":2,"disp":true,"width":"10","collapse":true},{"id":"b123e6c8.d3e948","type":"ui_tab","z":"","name":"Slide","icon":"dashboard","order":2,"disabled":false,"hidden":false}]

[EDIT]

krambriw · 11 March 2020 16:29

Nice, and this is my suggestion, just got it working as well!

[{"id":"8a69170a.2a6d78","type":"ui_template","z":"2d489fd9.1eedd","group":"31b5112.2d60fee","name":"highlightPict","order":2,"width":9,"height":6,"format":"<!DOCTYPE html>\n<html>\n<style type=\"text/css\"> \n.wrapper {\n    position:absolute;\n    width:320px;\n    height:240px;\n    border:2px solid grey;\n}\n\n.box {\n    position:absolute;\n    top:{{msg.y}}px;\n    left:{{msg.x}}px;\n    width:{{msg.width}}px;\n    height:{{msg.height}}px;\n    border:2px solid red;\n    background-color:transparent\n    z-index: 2;\n}\n</style> \n\n<script type=\"text/javascript\">\nscope.$watch('msg', function(msg) {\n    inMessage(msg);\n});\n\nfunction inMessage(event) {\n    if (event.payload) {\n        //image = event.payload;\n        //alert(image);\n        document.getElementById('cX').src = \"data:image/jpg;base64,\"+event.payload;\n\n    }\n}\n\n</script>\n\n\n<body>\n\n\n<div class=\"wrapper\">\n     <img src=\"\" id=\"cX\" />\n     <div class=\"box\"></div>\n</div>\n</body>\n</html>\n\n\n\n<!--\n//////////////////////////////////////////////////////////////////////////////\n<!DOCTYPE html>\n<html>\n<head>\n<style> \n#example1 {\n  background-image: url(data:image/jpg;base64,{{msg.payload}});\n  border: 1px solid blue;  \n  background-size:auto 100% ;\n  background-repeat: no-repeat;\n  /*\n}\nhttp://newscrane.com/wp-content/uploads/2019/09/Car-insurance-Newscrane-02.jpg\nsvg {\n border: 1px solid green;\n}\n*/\n</style>\n</head>\n<body>\n<div id=\"example1\">\n<svg width=\"320\" height=\"240\" style=\"border: 1px solid lightgrey;\">\n  <rect x={{msg.x}} y={{msg.y}} width={{msg.width}} height={{msg.height}} style=\"stroke:red;stroke-width:2;fill-opacity:0\" />\n  Sorry, your browser does not support inline SVG.  \n</svg>\n</div>\n</body>\n</html>\n-->\n<!--\n<img width=\"100%\" height=\"100%\" alt=\"HighLight Picture\" src=\"data:image/png;base64,{{msg.payload}}\" />\n<span style=\"font-size:10px;\">{{msg.fileName}} {{msg.affSlideIndex}}/{{msg.numSlides}}</span> \n-->\n","storeOutMessages":true,"fwdInMessages":true,"templateScope":"local","x":1140,"y":410,"wires":[[]]},{"id":"2fccc750.eba658","type":"tensorflowCoco","z":"2d489fd9.1eedd","name":"","model":"","scoreThreshold":"","x":560,"y":510,"wires":[["efbd6894.4216b8","8f62009e.983cb","f37f4cb9.3def6","99bb537b.b3b73"]]},{"id":"efbd6894.4216b8","type":"debug","z":"2d489fd9.1eedd","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":720,"y":570,"wires":[]},{"id":"8f62009e.983cb","type":"debug","z":"2d489fd9.1eedd","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"classes","targetType":"msg","x":740,"y":540,"wires":[]},{"id":"2085ac81.9563d4","type":"base64","z":"2d489fd9.1eedd","name":"","action":"str","property":"payload","x":560,"y":410,"wires":[["1dd54700.7468b9","8a69170a.2a6d78"]]},{"id":"f37f4cb9.3def6","type":"ui_table","z":"2d489fd9.1eedd","group":"31b5112.2d60fee","name":"table detection","order":8,"width":10,"height":2,"columns":[],"outputs":0,"cts":false,"x":750,"y":480,"wires":[]},{"id":"2ac71df3.5060f2","type":"http request","z":"2d489fd9.1eedd","name":"","method":"GET","ret":"bin","paytoqs":false,"url":"https://loremflickr.com/320/240/sport","tls":"","persist":false,"proxy":"","authType":"","x":370,"y":470,"wires":[["2fccc750.eba658","2085ac81.9563d4"]]},{"id":"dfc95a0d.0305c8","type":"ui_button","z":"2d489fd9.1eedd","name":"","group":"31b5112.2d60fee","order":6,"width":1,"height":1,"passthru":true,"label":"New Picture","tooltip":"","color":"","bgcolor":"","icon":"","payload":"","payloadType":"str","topic":"","x":190,"y":520,"wires":[["2ac71df3.5060f2"]]},{"id":"9031a37c.91ff","type":"function","z":"2d489fd9.1eedd","name":"posiRectangle","func":"//let pictureB64[] = msg.payload;\nif(msg.detect.length>0){\n    let box = msg.detect[0];\n    msg.x = box.bbox[0];\n    msg.y = box.bbox[1];\n    msg.width = box.bbox[2];\n    msg.height= box.bbox[3];\n    msg.class = box[\"class\"];\n    msg.score = box[\"score\"];\n    \n    return msg;\n}\n\n\n\n\n\n/*\n252x200 : person devant chalet\n0: object\n    bbox: array[4]\n        0: 127\n        1: 86\n        2: 29\n        3: 90\n    class: \"person\"\n    score: 0.9015087080001831\n    \n    x=127 y=86 width=127-90  height=86-29\n*/","outputs":1,"noerr":0,"x":920,"y":510,"wires":[["bbd8edb4.45372","8a69170a.2a6d78"]]},{"id":"bbd8edb4.45372","type":"debug","z":"2d489fd9.1eedd","name":"rectangle","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":1090,"y":510,"wires":[]},{"id":"fba73baf.e43ef8","type":"inject","z":"2d489fd9.1eedd","name":"test","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":140,"y":430,"wires":[["dfc95a0d.0305c8"]]},{"id":"99bb537b.b3b73","type":"change","z":"2d489fd9.1eedd","name":"pay to detect","rules":[{"t":"move","p":"payload","pt":"msg","to":"detect","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":740,"y":510,"wires":[["9031a37c.91ff"]]},{"id":"1dd54700.7468b9","type":"debug","z":"2d489fd9.1eedd","name":"b64","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":780,"y":380,"wires":[]},{"id":"31b5112.2d60fee","type":"ui_group","z":"","name":"detection","tab":"da82a0d.14db76","order":2,"disp":true,"width":"10","collapse":true},{"id":"da82a0d.14db76","type":"ui_tab","z":"","name":"Slide","icon":"dashboard","order":2,"disabled":false,"hidden":false}]

SuperNinja · 11 March 2020 16:37

great, another variation
now you should be able to display the other objects present, at least the 2 with the best score

krambriw · 11 March 2020 17:39

Just a sample, displaying detected objects one-by-one

(This one is also supporting images received via mqtt, you can just delete that part)

[{"id":"379c521a.e9d72e","type":"comment","z":"2d489fd9.1eedd","name":"NR Object detection","info":"","x":170,"y":40,"wires":[]},{"id":"8a69170a.2a6d78","type":"ui_template","z":"2d489fd9.1eedd","group":"31b5112.2d60fee","name":"highlightPict","order":2,"width":"18","height":"12","format":"<!DOCTYPE html>\n<html>\n<style type=\"text/css\"> \n.wrapper {\n    position:absolute;\n    width:320px;\n    height:240px;\n    border:2px solid grey;\n}\n\n.box {\n    position:absolute;\n    top:{{msg.y}}px;\n    left:{{msg.x}}px;\n    width:{{msg.width}}px;\n    height:{{msg.height}}px;\n    border:2px solid red;\n    background-color:transparent\n    z-index: 2;\n}\n.text {\n    position:absolute;\n    top:{{msg.y+5}}px;\n    left: {{msg.x+5}}px;\n    color: red;\n\n}\n</style> \n\n<script type=\"text/javascript\">\nscope.$watch('msg', function(msg) {\n    inMessage(msg);\n});\n\nfunction inMessage(event) {\n    if (event.payload) {\n        document.getElementById('cX').src = \"data:image/jpg;base64,\"+event.payload;\n    }\n}\n\n</script>\n\n\n<body>\n\n\n<div class=\"wrapper\">\n     <img src=\"\" id=\"cX\" />\n     <div class=\"box\"></div>\n     <div class=\"text\">{{msg.class}}</div>\n</div>\n</body>\n</html>\n\n\n\n","storeOutMessages":true,"fwdInMessages":true,"templateScope":"local","x":1130,"y":60,"wires":[[]]},{"id":"2fccc750.eba658","type":"tensorflowCoco","z":"2d489fd9.1eedd","name":"","model":"","scoreThreshold":"","x":570,"y":240,"wires":[["efbd6894.4216b8","8f62009e.983cb","f37f4cb9.3def6","99bb537b.b3b73","9a7b8548.3747c8"]]},{"id":"efbd6894.4216b8","type":"debug","z":"2d489fd9.1eedd","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":820,"y":310,"wires":[]},{"id":"8f62009e.983cb","type":"debug","z":"2d489fd9.1eedd","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"classes","targetType":"msg","x":840,"y":260,"wires":[]},{"id":"2085ac81.9563d4","type":"base64","z":"2d489fd9.1eedd","name":"","action":"str","property":"payload","x":560,"y":60,"wires":[["8a69170a.2a6d78"]]},{"id":"f37f4cb9.3def6","type":"ui_table","z":"2d489fd9.1eedd","group":"31b5112.2d60fee","name":"table detection","order":8,"width":10,"height":2,"columns":[],"outputs":0,"cts":false,"x":850,"y":160,"wires":[]},{"id":"2ac71df3.5060f2","type":"http request","z":"2d489fd9.1eedd","name":"","method":"GET","ret":"bin","paytoqs":false,"url":"https://loremflickr.com/320/240/sport","tls":"","persist":false,"proxy":"","authType":"","x":350,"y":140,"wires":[["2fccc750.eba658","2085ac81.9563d4","9a7b8548.3747c8"]]},{"id":"dfc95a0d.0305c8","type":"ui_button","z":"2d489fd9.1eedd","name":"","group":"31b5112.2d60fee","order":6,"width":1,"height":1,"passthru":true,"label":"New Picture","tooltip":"","color":"","bgcolor":"","icon":"","payload":"","payloadType":"str","topic":"","x":150,"y":140,"wires":[["2ac71df3.5060f2"]]},{"id":"9031a37c.91ff","type":"function","z":"2d489fd9.1eedd","name":"posiRectangle","func":"if(msg.detect.length>0){\n    msg.detect.forEach(element => writeBox(element))\n}\n\nfunction writeBox(element) {\n    let box = element;\n    msg.x = box.bbox[0];\n    msg.y = box.bbox[1];\n    msg.width = box.bbox[2];\n    msg.height= box.bbox[3];\n    msg.class = box[\"class\"];\n    msg.score = box[\"score\"];\n    node.send(msg);\n}\n\n\n","outputs":1,"noerr":0,"x":1030,"y":210,"wires":[["bbd8edb4.45372","ac45a3a3.d45b5"]]},{"id":"bbd8edb4.45372","type":"debug","z":"2d489fd9.1eedd","name":"rectangle","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":1240,"y":210,"wires":[]},{"id":"fba73baf.e43ef8","type":"inject","z":"2d489fd9.1eedd","name":"test","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":130,"y":80,"wires":[["dfc95a0d.0305c8"]]},{"id":"99bb537b.b3b73","type":"change","z":"2d489fd9.1eedd","name":"pay to detect","rules":[{"t":"move","p":"payload","pt":"msg","to":"detect","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":840,"y":210,"wires":[["9031a37c.91ff"]]},{"id":"9a7b8548.3747c8","type":"function","z":"2d489fd9.1eedd","name":"","func":"var time = new Date().getTime()\nnode.warn(time)\n","outputs":0,"noerr":0,"x":570,"y":140,"wires":[]},{"id":"3bf60ae8.caf4f6","type":"mqtt in","z":"2d489fd9.1eedd","name":"","topic":"image/#","qos":"2","datatype":"auto","broker":"2a019090.5ba4d","x":160,"y":240,"wires":[["adb4835c.d0472","1804351d.75eb0b"]]},{"id":"adb4835c.d0472","type":"switch","z":"2d489fd9.1eedd","name":"","property":"payload","propertyType":"msg","rules":[{"t":"istype","v":"buffer","vt":"buffer"}],"checkall":"true","repair":false,"outputs":1,"x":370,"y":240,"wires":[["2085ac81.9563d4","9a7b8548.3747c8","2fccc750.eba658"]]},{"id":"1804351d.75eb0b","type":"debug","z":"2d489fd9.1eedd","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":360,"y":310,"wires":[]},{"id":"ac45a3a3.d45b5","type":"delay","z":"2d489fd9.1eedd","name":"","pauseType":"rate","timeout":"5","timeoutUnits":"seconds","rate":"1","nbRateUnits":"3","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"x":1090,"y":130,"wires":[["8a69170a.2a6d78"]]},{"id":"31b5112.2d60fee","type":"ui_group","z":"","name":"detection","tab":"da82a0d.14db76","order":2,"disp":true,"width":"18","collapse":true},{"id":"2a019090.5ba4d","type":"mqtt-broker","z":"","name":"","broker":"192.168.0.240","port":"1883","clientid":"","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","closeTopic":"","closePayload":"","willTopic":"","willQos":"0","willPayload":""},{"id":"da82a0d.14db76","type":"ui_tab","z":"","name":"Slide","icon":"dashboard","order":2,"disabled":false,"hidden":false}]

dceejay · 11 March 2020 18:48

Loving what you guys are doing !

SuperNinja · 11 March 2020 19:50

nice add !

SuperNinja · 11 March 2020 19:53

Thank you for creating this node @dceejay !
Do you have any idea why your node is not working on WIN10 (see error log at the start of the post)? The bad luck ?

krambriw · 11 March 2020 20:30

I know, this was for @dceejay but I could not keep my mouth shut; I believe it is because the javascrript version tensorflow (tfjs) is not supported on your platform. I got this error message on the Nvidia Jetson Nano when I tried installing

npm install @tensorflow/tfjs-node-gpu

GPU-linux-1.7.0.tar.gz
* Downloading libtensorflow
(node:9011) UnhandledPromiseRejectionWarning: Error: Unsupported system: gpu-linux-arm64

SuperNinja · 11 March 2020 20:44

oh come on
yep, I saw that shows you a plug-in error on your platform, but me it says

"Error: Cannot find module ..."

Can we conclude that the platform is not compatible or do I have a module to add? so which one?

zenofmud · 11 March 2020 21:00

hmmm got the same thing trying to install it on a Pi Zero W (just thought I'd try) though mine ends with Error: Unsupported system: gpu-linux-arm

SuperNinja · 11 March 2020 21:17

So for now, the only hardware that works is the RPI 3B (for me).
Surely the RPI 3B + ...

zenofmud · 11 March 2020 22:48

I have it working on a RPI 3B+

Topic		Replies	Views
Any suggestions on how to do this with Node-red and TensorFlow.js. General	3	546	30 October 2020
Detecting objects using ESP32 CAM and NODE RED General	50	6320	18 September 2021
Tf coco ssd don't work General	21	1015	28 June 2022
Object detection with predefined set and DAHUA IP-CAM General	7	150	24 September 2024
Update to tfjs-coco-ssd node (added bounding boxes) General	3	559	27 October 2020

Object Detection using node-red-contrib-tfjs-coco-ssd

[EDIT] Compatible material

Related topics