Object Detection using node-red-contrib-tfjs-coco-ssd

General
1

Hi all ,
i try to use @dceejay node-red-contrib-tfjs-coco-ssd to detect object in picture with the example attached :
image

[{"id":"a2e68203.7be26","type":"tensorflowCoco","z":"d0cece46.87ac6","name":"","model":"","x":510,"y":1160,"wires":[["f7c2654b.8582b8","92bee663.a421b8","19ca5cc7.3b49d3"]]},{"id":"f7c2654b.8582b8","type":"debug","z":"d0cece46.87ac6","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":810,"y":1200,"wires":[]},{"id":"92bee663.a421b8","type":"debug","z":"d0cece46.87ac6","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"classes","targetType":"msg","x":810,"y":1240,"wires":[]},{"id":"69f3ad28.a4b994","type":"base64","z":"d0cece46.87ac6","name":"","action":"str","property":"payload","x":500,"y":1100,"wires":[["a4ad37b6.b74438"]]},{"id":"a4ad37b6.b74438","type":"template","z":"d0cece46.87ac6","name":"","field":"payload","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"<img width=\"320px\" height=\"240px\" src=\"data:image/jpg;base64,{{{payload}}}\">","output":"str","x":640,"y":1100,"wires":[["b54432f0.8059a"]]},{"id":"b54432f0.8059a","type":"ui_template","z":"d0cece46.87ac6","group":"aef761dd.7e9dd","name":"","order":2,"width":7,"height":5,"format":"<div ng-bind-html=\"msg.payload\"></div>","storeOutMessages":true,"fwdInMessages":true,"templateScope":"local","x":800,"y":1100,"wires":[[]]},{"id":"19ca5cc7.3b49d3","type":"ui_table","z":"d0cece46.87ac6","group":"aef761dd.7e9dd","name":"","order":15,"width":9,"height":3,"columns":[],"outputs":0,"cts":false,"x":790,"y":1160,"wires":[]},{"id":"dd9cd04b.d370a","type":"http request","z":"d0cece46.87ac6","name":"","method":"GET","ret":"bin","paytoqs":false,"url":"https://loremflickr.com/320/240/sport","tls":"","persist":false,"proxy":"","authType":"","x":330,"y":1100,"wires":[["a2e68203.7be26","69f3ad28.a4b994"]]},{"id":"9c31f01c.dd7c4","type":"ui_button","z":"d0cece46.87ac6","name":"","group":"aef761dd.7e9dd","order":13,"width":7,"height":1,"passthru":false,"label":"New Picture","tooltip":"","color":"","bgcolor":"","icon":"","payload":"","payloadType":"str","topic":"","x":170,"y":1100,"wires":[["dd9cd04b.d370a"]]},{"id":"aef761dd.7e9dd","type":"ui_group","z":"","name":"Object detection","tab":"3dce94d.cb4316c","disp":true,"width":"9","collapse":false},{"id":"3dce94d.cb4316c","type":"ui_tab","z":"","name":"Home","icon":"dashboard"}]

The readme is like this :

A Node-RED node for Object Detection using TensorFlowJS CoCo SSD.

NOTE : Tensorflow.js is only available on certain OS/Hardware/processor combinations. It does not autimatically install/work on all platforms. You may be unlucky.

The installation with the palette went well with the addition of the ui-table node. But I have this error:
Error: Cannot find module 'C:\Users\christian\.Node-RED-Desktop\node_modules\@tensorflow\tfjs-node\lib\napi-v4\tfjs_binding.node'Require stack:- C:\Users\christian\.Node-RED-Desktop\node_modules\@tensorflow\tfjs-node\dist\index.js- C:\Users\christian\.Node-RED-Desktop\node_modules\node-red-contrib-tfjs-coco-ssd\tfjs.js- C:\Users\christian\AppData\Local\Programs\node-red-desktop\resources\app.asar\node_modules\@node-red\registry\lib\loader.js- C:\Users\christian\AppData\Local\Programs\node-red-desktop\resources\app.asar\node_modules\@node-red\registry\lib\index.js- C:\Users\christian\AppData\Local\Programs\node-red-desktop\resources\app.asar\node_modules\@node-red\runtime\lib\nodes\index.js- C:\Users\christian\AppData\Local\Programs\node-red-desktop\resources\app.asar\node_modules\@node-red\runtime\lib\index.js- C:\Users\christian\AppData\Local\Programs\node-red-desktop\resources\app.asar\main\node-red.js- C:\Users\christian\AppData\Local\Programs\node-red-desktop\resources\app.asar\main\config-manager.js- C:\Users\christian\AppData\Local\Programs\node-red-desktop\resources\app.asar\main\main.js-


Is something missing from the installation, or am I just "out of luck"?
Should I install node-red-contrib-tfjs? I can't find it in the manage palette.

P.S. I use Node-Red Desktop on Win10 64Bits (NR version v1.0.3, Node.js v12.8.1)

[EDIT] Compatible material

To date, here is the list of devices compatible and not, with this node (this list will be updated according to future testers) :

  • Raspberry PI 3B and 3B+
  • Mac
  • Lenovo with Debian
  • WIN10 PC with nodejs : v12.16.1 & npm : 6.13.4

Unsupported hardware (at the time of this writing)

  • Pi Zero W
  • NVIDIA jetson Nano
1 Like
2

Nice!!!
I did just install the node (and the ui_table that I didn't have installed already)

image
image534Ă—609 82.8 KB

3

Very impressive I must say. On a Pi3 it takes just around 1000 ms for the node to analyze the picture for objects. That is by far the best I have seen on a Pi!!!

The node also supports pictures delivered via mqtt. Next, I have to try this on one of my Nvidia Jetson Nano where I have the GPU waiting for this
EDIT: This failed, same error as Christian (@SuperNinja) reported above, seems not to be compatible with the Nvidia platform

Challenge, would be nice to have a rectangle overlayed in the picture representing the box coordinates

4

This is what i'm working on :yum:! Screenshot_20200311-075403|690x388, 50%

5

:clap:

Very nice!!!

Myself, trying to figure out how to make this work with a GPU. I have the Nvidia stuff and expect it should increase performance rather significant (if I will figure out).

Tried it already also with an old laptop running debian, analyze is ready in approx 200 ms

1 Like
6

Challenge : rectangle overlayed in the picture : NEED HELP !
In this test i display only the first class

image
image841Ă—451 118 KB

image
As you can see the "person" is detected, the box displayed BUT the picture flash 1ms and disapear !

[{"id":"24bd5862.38b1b8","type":"ui_template","z":"a8b1b208.7036f","group":"7c26c8d0.c1e658","name":"highlightPict","order":2,"width":9,"height":6,"format":"<!DOCTYPE html>\n<html>\n<style type=\"text/css\"> \n.wrapper {\n    position:relative;\n    width:320px;\n    height:240px;\n    border:2px solid grey;\n}\n\n.box {\n    position:absolute;\n    top:{{msg.y}}px;\n    left:{{msg.x}}px;\n    width:{{msg.width}}px;\n    height:{{msg.height}}px;\n    border:2px solid red;\n    background-color:transparent\n}\n </style> \n<body>\n\n\n<div class=\"wrapper\">\n     <img src=\"data:image/jpg;base64,{{msg.payload}}\"  />\n     <div class=\"box\"></div>\n</div>\n</body>\n</html>\n\n\n\n<!--\n//////////////////////////////////////////////////////////////////////////////\n<!DOCTYPE html>\n<html>\n<head>\n<style> \n#example1 {\n  background-image: url(data:image/jpg;base64,{{msg.payload}});\n  border: 1px solid blue;  \n  background-size:auto 100% ;\n  background-repeat: no-repeat;\n  /*\n}\nhttp://newscrane.com/wp-content/uploads/2019/09/Car-insurance-Newscrane-02.jpg\nsvg {\n border: 1px solid green;\n}\n*/\n</style>\n</head>\n<body>\n<div id=\"example1\">\n<svg width=\"320\" height=\"240\" style=\"border: 1px solid lightgrey;\">\n  <rect x={{msg.x}} y={{msg.y}} width={{msg.width}} height={{msg.height}} style=\"stroke:red;stroke-width:2;fill-opacity:0\" />\n  Sorry, your browser does not support inline SVG.  \n</svg>\n</div>\n</body>\n</html>\n-->\n<!--\n<img width=\"100%\" height=\"100%\" alt=\"HighLight Picture\" src=\"data:image/png;base64,{{msg.payload}}\" />\n<span style=\"font-size:10px;\">{{msg.fileName}} {{msg.affSlideIndex}}/{{msg.numSlides}}</span> \n-->\n","storeOutMessages":true,"fwdInMessages":true,"templateScope":"local","x":1120,"y":1800,"wires":[[]]},{"id":"86f3292b.6471b8","type":"tensorflowCoco","z":"a8b1b208.7036f","name":"","model":"","scoreThreshold":"","x":590,"y":1840,"wires":[["2665fee8.ccc712","5b1ba3a2.77ebdc","61d7c39d.c5efdc","acbb43ee.e63df","c2a407c5.827fb8"]]},{"id":"2665fee8.ccc712","type":"debug","z":"a8b1b208.7036f","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":770,"y":1900,"wires":[]},{"id":"5b1ba3a2.77ebdc","type":"debug","z":"a8b1b208.7036f","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"classes","targetType":"msg","x":770,"y":1870,"wires":[]},{"id":"f2874676.b4f3b8","type":"base64","z":"a8b1b208.7036f","name":"","action":"str","property":"payload","x":580,"y":1770,"wires":[["d215245.d4919d8","24bd5862.38b1b8"]]},{"id":"61d7c39d.c5efdc","type":"ui_table","z":"a8b1b208.7036f","group":"7c26c8d0.c1e658","name":"table detection","order":8,"width":10,"height":2,"columns":[],"outputs":0,"cts":false,"x":780,"y":1810,"wires":[]},{"id":"bf528beb.36b538","type":"http request","z":"a8b1b208.7036f","name":"","method":"GET","ret":"bin","paytoqs":false,"url":"https://loremflickr.com/320/240/sport","tls":"","persist":false,"proxy":"","authType":"","x":370,"y":1850,"wires":[["86f3292b.6471b8","f2874676.b4f3b8","97a97ff2.45acd","64c32917.ff8b28"]]},{"id":"fd369cc7.f262b","type":"ui_button","z":"a8b1b208.7036f","name":"","group":"7c26c8d0.c1e658","order":6,"width":1,"height":1,"passthru":true,"label":"New Picture","tooltip":"","color":"","bgcolor":"","icon":"","payload":"","payloadType":"str","topic":"","x":220,"y":1850,"wires":[["bf528beb.36b538"]]},{"id":"64c32917.ff8b28","type":"image-info","z":"a8b1b208.7036f","name":"","x":590,"y":1910,"wires":[[]]},{"id":"27cb047f.a8173c","type":"function","z":"a8b1b208.7036f","name":"posiRectangle","func":"//let pictureB64[] = msg.payload;\nlet box = msg.detect[0];\nmsg.x = box.bbox[0];\nmsg.y = box.bbox[1];\nmsg.width = box.bbox[2];\nmsg.height= box.bbox[3];\nmsg.class = box[\"class\"];\nmsg.score = box[\"score\"];\n\nreturn msg;\n\n\n\n\n\n/*\n252x200 : person devant chalet\n0: object\n    bbox: array[4]\n        0: 127\n        1: 86\n        2: 29\n        3: 90\n    class: \"person\"\n    score: 0.9015087080001831\n    \n    x=127 y=86 width=127-90  height=86-29\n*/","outputs":1,"noerr":0,"x":930,"y":1840,"wires":[["91b9bccd.feaa9","24bd5862.38b1b8"]]},{"id":"91b9bccd.feaa9","type":"debug","z":"a8b1b208.7036f","name":"rectangle","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":1110,"y":1840,"wires":[]},{"id":"97a97ff2.45acd","type":"image","z":"a8b1b208.7036f","name":"","width":"400","data":"payload","dataType":"msg","thumbnail":false,"active":true,"x":600,"y":1980,"wires":[]},{"id":"5f00cc2e.197f64","type":"inject","z":"a8b1b208.7036f","name":"test","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":80,"y":1850,"wires":[["fd369cc7.f262b"]]},{"id":"c2a407c5.827fb8","type":"change","z":"a8b1b208.7036f","name":"pay to detect","rules":[{"t":"move","p":"payload","pt":"msg","to":"detect","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":770,"y":1840,"wires":[["27cb047f.a8173c"]]},{"id":"d215245.d4919d8","type":"debug","z":"a8b1b208.7036f","name":"b64","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":890,"y":1720,"wires":[]},{"id":"7c26c8d0.c1e658","type":"ui_group","z":"","name":"detection","tab":"b123e6c8.d3e948","order":2,"disp":true,"width":"10","collapse":true},{"id":"b123e6c8.d3e948","type":"ui_tab","z":"","name":"Slide","icon":"dashboard","order":2,"disabled":false,"hidden":false}]

In an other hand , there is "nothing" to detect :
image
Detect nothing = the picture is loaded

This is the template node :

<!DOCTYPE html>
<html>
<style type="text/css"> 
.wrapper {
    position:relative;
    width:320px;
    height:240px;
    border:2px solid grey;
}

.box {
    position:absolute;
    top:{{msg.y}}px;
    left:{{msg.x}}px;
    width:{{msg.width}}px;
    height:{{msg.height}}px;
    border:2px solid red;
    background-color:transparent
}
 </style> 
<body>


<div class="wrapper">
     <img src="data:image/jpg;base64,{{msg.payload}}"  />
     <div class="box"></div>
</div>
</body>
</html>

I would like the image to persist, before displaying the box! How to do ?

7

The problem here is that the image is overwritten by the msg from the posiRectangle. Somehow the image needs to be "saved" in the ui template and then used in the html code when the box data comes in

I tested a bit by instead saving the image to file and then load it in the html when the box data arrives and then it works but this is of course not a good way doing it

image
image534Ă—609 72 KB

8

not bad !!
that's for sure, it saves an image on the SD card, it's not good for its life.
We are not far from the goal, it would be nice to be able to combine the image + the box in a file and then display it in the HTML code
Is it possible to put in a function, the msg.payload of the image + msg.x, msg.y, msg.width ... so as to send everything at once? It will avoid overwriting the image on arrival of msg.x, y, ...

9

It's alive !!! :star_struck:
image

[{"id":"86f3292b.6471b8","type":"tensorflowCoco","z":"a8b1b208.7036f","name":"","model":"","scoreThreshold":"","x":590,"y":1840,"wires":[["2665fee8.ccc712","5b1ba3a2.77ebdc","61d7c39d.c5efdc","c2a407c5.827fb8"]]},{"id":"2665fee8.ccc712","type":"debug","z":"a8b1b208.7036f","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":770,"y":1900,"wires":[]},{"id":"5b1ba3a2.77ebdc","type":"debug","z":"a8b1b208.7036f","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"classes","targetType":"msg","x":770,"y":1870,"wires":[]},{"id":"f2874676.b4f3b8","type":"base64","z":"a8b1b208.7036f","name":"","action":"str","property":"payload","x":580,"y":1770,"wires":[["d215245.d4919d8","cc75d53b.4b47b8"]]},{"id":"61d7c39d.c5efdc","type":"ui_table","z":"a8b1b208.7036f","group":"7c26c8d0.c1e658","name":"table detection","order":8,"width":10,"height":2,"columns":[],"outputs":0,"cts":false,"x":780,"y":1810,"wires":[]},{"id":"bf528beb.36b538","type":"http request","z":"a8b1b208.7036f","name":"","method":"GET","ret":"bin","paytoqs":false,"url":"https://loremflickr.com/320/240/sport","tls":"","persist":false,"proxy":"","authType":"","x":370,"y":1850,"wires":[["fb2a6f1a.6c4ed"]]},{"id":"fd369cc7.f262b","type":"ui_button","z":"a8b1b208.7036f","name":"","group":"7c26c8d0.c1e658","order":6,"width":1,"height":1,"passthru":true,"label":"New Picture","tooltip":"","color":"","bgcolor":"","icon":"","payload":"","payloadType":"str","topic":"","x":220,"y":1850,"wires":[["bf528beb.36b538"]]},{"id":"64c32917.ff8b28","type":"image-info","z":"a8b1b208.7036f","name":"","x":590,"y":1910,"wires":[[]]},{"id":"27cb047f.a8173c","type":"function","z":"a8b1b208.7036f","name":"posiRectangle","func":"var pictureBuffer = flow.get('pictureBuffer')|| [];\nmsg.pictureBuffer = pictureBuffer;\n//let pictureB64[] = msg.payload;\nlet box = msg.detect[0];\nmsg.x = box.bbox[0];\nmsg.y = box.bbox[1];\nmsg.width = box.bbox[2];\nmsg.height= box.bbox[3];\nmsg.class = box[\"class\"];\nmsg.score = box[\"score\"];\n\nreturn msg;\n\n\n\n\n\n/*\n252x200 : person devant chalet\n0: object\n    bbox: array[4]\n        0: 127\n        1: 86\n        2: 29\n        3: 90\n    class: \"person\"\n    score: 0.9015087080001831\n    \n    x=127 y=86 width=127-90  height=86-29\n*/","outputs":1,"noerr":0,"x":950,"y":1840,"wires":[["91b9bccd.feaa9","4d7ff98d.d0a008"]]},{"id":"91b9bccd.feaa9","type":"debug","z":"a8b1b208.7036f","name":"rectangle","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":1110,"y":1840,"wires":[]},{"id":"97a97ff2.45acd","type":"image","z":"a8b1b208.7036f","name":"","width":"400","data":"payload","dataType":"msg","thumbnail":false,"active":true,"x":600,"y":1980,"wires":[]},{"id":"5f00cc2e.197f64","type":"inject","z":"a8b1b208.7036f","name":"test","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":80,"y":1850,"wires":[["fd369cc7.f262b"]]},{"id":"c2a407c5.827fb8","type":"change","z":"a8b1b208.7036f","name":"pay to detect","rules":[{"t":"move","p":"payload","pt":"msg","to":"detect","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":770,"y":1840,"wires":[["27cb047f.a8173c"]]},{"id":"d215245.d4919d8","type":"debug","z":"a8b1b208.7036f","name":"b64","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":890,"y":1720,"wires":[]},{"id":"66276b4a.5c6cb4","type":"link in","z":"a8b1b208.7036f","name":"coco","links":["fb2a6f1a.6c4ed","39e7bc7c.c0a794","308e49b6.df1dc6"],"x":455,"y":1940,"wires":[["f2874676.b4f3b8","97a97ff2.45acd","64c32917.ff8b28","86f3292b.6471b8"]]},{"id":"fb2a6f1a.6c4ed","type":"link out","z":"a8b1b208.7036f","name":"","links":["66276b4a.5c6cb4"],"x":320,"y":1940,"wires":[]},{"id":"4d7ff98d.d0a008","type":"ui_template","z":"a8b1b208.7036f","group":"7c26c8d0.c1e658","name":"highlightPict","order":2,"width":9,"height":6,"format":"<!DOCTYPE html>\n<html>\n<style type=\"text/css\"> \n.wrapper {\n    position:relative;\n    width:320px;\n    height:240px;\n    border:2px solid grey;\n}\n\n.box {\n    position:absolute;\n    top:{{msg.y}}px;\n    left:{{msg.x}}px;\n    width:{{msg.width}}px;\n    height:{{msg.height}}px;\n    border:2px solid red;\n    background-color:transparent\n}\n\n.text {\n    position:absolute;\n    top:{{msg.y}}px;\n    left: {{msg.x}}px;\n    color: red;\n\n}\n </style> \n<body>\n\n\n<div class=\"wrapper\">\n    <img src=\"data:image/jpg;base64,{{msg.pictureBuffer}}\"/> \n     <div class=\"box\"></div>\n     <div class=\"text\">{{msg.class}}</div>\n     \n</div>\n</body>\n</html>\n <!-- \n\n-->\n<!--\n//////////////////////////////////////////////////////////////////////////////\n<!DOCTYPE html>\n<html>\n<head>\n<style> \n#example1 {\n  background-image: url(data:image/jpg;base64,{{msg.payload}});\n  border: 1px solid blue;  \n  background-size:auto 100% ;\n  background-repeat: no-repeat;\n  /*\n}\nhttp://newscrane.com/wp-content/uploads/2019/09/Car-insurance-Newscrane-02.jpg\nsvg {\n border: 1px solid green;\n}\n*/\n</style>\n</head>\n<body>\n<div id=\"example1\">\n<svg width=\"320\" height=\"240\" style=\"border: 1px solid lightgrey;\">\n  <rect x={{msg.x}} y={{msg.y}} width={{msg.width}} height={{msg.height}} style=\"stroke:red;stroke-width:2;fill-opacity:0\" />\n  Sorry, your browser does not support inline SVG.  \n</svg>\n</div>\n</body>\n</html>\n-->\n<!--\n<img width=\"100%\" height=\"100%\" alt=\"HighLight Picture\" src=\"data:image/png;base64,{{msg.payload}}\" />\n<span style=\"font-size:10px;\">{{msg.fileName}} {{msg.affSlideIndex}}/{{msg.numSlides}}</span> \n-->\n","storeOutMessages":true,"fwdInMessages":true,"templateScope":"local","x":1120,"y":1760,"wires":[[]]},{"id":"cc75d53b.4b47b8","type":"function","z":"a8b1b208.7036f","name":"flow.set pictureBuffer","func":"// Créé une variable pour stocker le pictureBuffer si inexistante\nvar pictureBuffer = flow.get('pictureBuffer')|| [];\npictureBuffer = msg.payload;\n// store the value \nflow.set('pictureBuffer',pictureBuffer);\n\n\n\n\n\n","outputs":0,"noerr":0,"x":680,"y":1690,"wires":[]},{"id":"7c26c8d0.c1e658","type":"ui_group","z":"","name":"detection","tab":"b123e6c8.d3e948","order":2,"disp":true,"width":"10","collapse":true},{"id":"b123e6c8.d3e948","type":"ui_tab","z":"","name":"Slide","icon":"dashboard","order":2,"disabled":false,"hidden":false}]

[EDIT]

image
image846Ă—455 89.3 KB

10

Nice, and this is my suggestion, just got it working as well!

image
image1217Ă—259 34.8 KB 

[{"id":"8a69170a.2a6d78","type":"ui_template","z":"2d489fd9.1eedd","group":"31b5112.2d60fee","name":"highlightPict","order":2,"width":9,"height":6,"format":"<!DOCTYPE html>\n<html>\n<style type=\"text/css\"> \n.wrapper {\n    position:absolute;\n    width:320px;\n    height:240px;\n    border:2px solid grey;\n}\n\n.box {\n    position:absolute;\n    top:{{msg.y}}px;\n    left:{{msg.x}}px;\n    width:{{msg.width}}px;\n    height:{{msg.height}}px;\n    border:2px solid red;\n    background-color:transparent\n    z-index: 2;\n}\n</style> \n\n<script type=\"text/javascript\">\nscope.$watch('msg', function(msg) {\n    inMessage(msg);\n});\n\nfunction inMessage(event) {\n    if (event.payload) {\n        //image = event.payload;\n        //alert(image);\n        document.getElementById('cX').src = \"data:image/jpg;base64,\"+event.payload;\n\n    }\n}\n\n</script>\n\n\n<body>\n\n\n<div class=\"wrapper\">\n     <img src=\"\" id=\"cX\" />\n     <div class=\"box\"></div>\n</div>\n</body>\n</html>\n\n\n\n<!--\n//////////////////////////////////////////////////////////////////////////////\n<!DOCTYPE html>\n<html>\n<head>\n<style> \n#example1 {\n  background-image: url(data:image/jpg;base64,{{msg.payload}});\n  border: 1px solid blue;  \n  background-size:auto 100% ;\n  background-repeat: no-repeat;\n  /*\n}\nhttp://newscrane.com/wp-content/uploads/2019/09/Car-insurance-Newscrane-02.jpg\nsvg {\n border: 1px solid green;\n}\n*/\n</style>\n</head>\n<body>\n<div id=\"example1\">\n<svg width=\"320\" height=\"240\" style=\"border: 1px solid lightgrey;\">\n  <rect x={{msg.x}} y={{msg.y}} width={{msg.width}} height={{msg.height}} style=\"stroke:red;stroke-width:2;fill-opacity:0\" />\n  Sorry, your browser does not support inline SVG.  \n</svg>\n</div>\n</body>\n</html>\n-->\n<!--\n<img width=\"100%\" height=\"100%\" alt=\"HighLight Picture\" src=\"data:image/png;base64,{{msg.payload}}\" />\n<span style=\"font-size:10px;\">{{msg.fileName}} {{msg.affSlideIndex}}/{{msg.numSlides}}</span> \n-->\n","storeOutMessages":true,"fwdInMessages":true,"templateScope":"local","x":1140,"y":410,"wires":[[]]},{"id":"2fccc750.eba658","type":"tensorflowCoco","z":"2d489fd9.1eedd","name":"","model":"","scoreThreshold":"","x":560,"y":510,"wires":[["efbd6894.4216b8","8f62009e.983cb","f37f4cb9.3def6","99bb537b.b3b73"]]},{"id":"efbd6894.4216b8","type":"debug","z":"2d489fd9.1eedd","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":720,"y":570,"wires":[]},{"id":"8f62009e.983cb","type":"debug","z":"2d489fd9.1eedd","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"classes","targetType":"msg","x":740,"y":540,"wires":[]},{"id":"2085ac81.9563d4","type":"base64","z":"2d489fd9.1eedd","name":"","action":"str","property":"payload","x":560,"y":410,"wires":[["1dd54700.7468b9","8a69170a.2a6d78"]]},{"id":"f37f4cb9.3def6","type":"ui_table","z":"2d489fd9.1eedd","group":"31b5112.2d60fee","name":"table detection","order":8,"width":10,"height":2,"columns":[],"outputs":0,"cts":false,"x":750,"y":480,"wires":[]},{"id":"2ac71df3.5060f2","type":"http request","z":"2d489fd9.1eedd","name":"","method":"GET","ret":"bin","paytoqs":false,"url":"https://loremflickr.com/320/240/sport","tls":"","persist":false,"proxy":"","authType":"","x":370,"y":470,"wires":[["2fccc750.eba658","2085ac81.9563d4"]]},{"id":"dfc95a0d.0305c8","type":"ui_button","z":"2d489fd9.1eedd","name":"","group":"31b5112.2d60fee","order":6,"width":1,"height":1,"passthru":true,"label":"New Picture","tooltip":"","color":"","bgcolor":"","icon":"","payload":"","payloadType":"str","topic":"","x":190,"y":520,"wires":[["2ac71df3.5060f2"]]},{"id":"9031a37c.91ff","type":"function","z":"2d489fd9.1eedd","name":"posiRectangle","func":"//let pictureB64[] = msg.payload;\nif(msg.detect.length>0){\n    let box = msg.detect[0];\n    msg.x = box.bbox[0];\n    msg.y = box.bbox[1];\n    msg.width = box.bbox[2];\n    msg.height= box.bbox[3];\n    msg.class = box[\"class\"];\n    msg.score = box[\"score\"];\n    \n    return msg;\n}\n\n\n\n\n\n/*\n252x200 : person devant chalet\n0: object\n    bbox: array[4]\n        0: 127\n        1: 86\n        2: 29\n        3: 90\n    class: \"person\"\n    score: 0.9015087080001831\n    \n    x=127 y=86 width=127-90  height=86-29\n*/","outputs":1,"noerr":0,"x":920,"y":510,"wires":[["bbd8edb4.45372","8a69170a.2a6d78"]]},{"id":"bbd8edb4.45372","type":"debug","z":"2d489fd9.1eedd","name":"rectangle","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":1090,"y":510,"wires":[]},{"id":"fba73baf.e43ef8","type":"inject","z":"2d489fd9.1eedd","name":"test","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":140,"y":430,"wires":[["dfc95a0d.0305c8"]]},{"id":"99bb537b.b3b73","type":"change","z":"2d489fd9.1eedd","name":"pay to detect","rules":[{"t":"move","p":"payload","pt":"msg","to":"detect","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":740,"y":510,"wires":[["9031a37c.91ff"]]},{"id":"1dd54700.7468b9","type":"debug","z":"2d489fd9.1eedd","name":"b64","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":780,"y":380,"wires":[]},{"id":"31b5112.2d60fee","type":"ui_group","z":"","name":"detection","tab":"da82a0d.14db76","order":2,"disp":true,"width":"10","collapse":true},{"id":"da82a0d.14db76","type":"ui_tab","z":"","name":"Slide","icon":"dashboard","order":2,"disabled":false,"hidden":false}]
2 Likes
11

great, another variation :+1:
now you should be able to display the other objects present, at least the 2 with the best score

12

Just a sample, displaying detected objects one-by-one

(This one is also supporting images received via mqtt, you can just delete that part)

[{"id":"379c521a.e9d72e","type":"comment","z":"2d489fd9.1eedd","name":"NR Object detection","info":"","x":170,"y":40,"wires":[]},{"id":"8a69170a.2a6d78","type":"ui_template","z":"2d489fd9.1eedd","group":"31b5112.2d60fee","name":"highlightPict","order":2,"width":"18","height":"12","format":"<!DOCTYPE html>\n<html>\n<style type=\"text/css\"> \n.wrapper {\n    position:absolute;\n    width:320px;\n    height:240px;\n    border:2px solid grey;\n}\n\n.box {\n    position:absolute;\n    top:{{msg.y}}px;\n    left:{{msg.x}}px;\n    width:{{msg.width}}px;\n    height:{{msg.height}}px;\n    border:2px solid red;\n    background-color:transparent\n    z-index: 2;\n}\n.text {\n    position:absolute;\n    top:{{msg.y+5}}px;\n    left: {{msg.x+5}}px;\n    color: red;\n\n}\n</style> \n\n<script type=\"text/javascript\">\nscope.$watch('msg', function(msg) {\n    inMessage(msg);\n});\n\nfunction inMessage(event) {\n    if (event.payload) {\n        document.getElementById('cX').src = \"data:image/jpg;base64,\"+event.payload;\n    }\n}\n\n</script>\n\n\n<body>\n\n\n<div class=\"wrapper\">\n     <img src=\"\" id=\"cX\" />\n     <div class=\"box\"></div>\n     <div class=\"text\">{{msg.class}}</div>\n</div>\n</body>\n</html>\n\n\n\n","storeOutMessages":true,"fwdInMessages":true,"templateScope":"local","x":1130,"y":60,"wires":[[]]},{"id":"2fccc750.eba658","type":"tensorflowCoco","z":"2d489fd9.1eedd","name":"","model":"","scoreThreshold":"","x":570,"y":240,"wires":[["efbd6894.4216b8","8f62009e.983cb","f37f4cb9.3def6","99bb537b.b3b73","9a7b8548.3747c8"]]},{"id":"efbd6894.4216b8","type":"debug","z":"2d489fd9.1eedd","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":820,"y":310,"wires":[]},{"id":"8f62009e.983cb","type":"debug","z":"2d489fd9.1eedd","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"classes","targetType":"msg","x":840,"y":260,"wires":[]},{"id":"2085ac81.9563d4","type":"base64","z":"2d489fd9.1eedd","name":"","action":"str","property":"payload","x":560,"y":60,"wires":[["8a69170a.2a6d78"]]},{"id":"f37f4cb9.3def6","type":"ui_table","z":"2d489fd9.1eedd","group":"31b5112.2d60fee","name":"table detection","order":8,"width":10,"height":2,"columns":[],"outputs":0,"cts":false,"x":850,"y":160,"wires":[]},{"id":"2ac71df3.5060f2","type":"http request","z":"2d489fd9.1eedd","name":"","method":"GET","ret":"bin","paytoqs":false,"url":"https://loremflickr.com/320/240/sport","tls":"","persist":false,"proxy":"","authType":"","x":350,"y":140,"wires":[["2fccc750.eba658","2085ac81.9563d4","9a7b8548.3747c8"]]},{"id":"dfc95a0d.0305c8","type":"ui_button","z":"2d489fd9.1eedd","name":"","group":"31b5112.2d60fee","order":6,"width":1,"height":1,"passthru":true,"label":"New Picture","tooltip":"","color":"","bgcolor":"","icon":"","payload":"","payloadType":"str","topic":"","x":150,"y":140,"wires":[["2ac71df3.5060f2"]]},{"id":"9031a37c.91ff","type":"function","z":"2d489fd9.1eedd","name":"posiRectangle","func":"if(msg.detect.length>0){\n    msg.detect.forEach(element => writeBox(element))\n}\n\nfunction writeBox(element) {\n    let box = element;\n    msg.x = box.bbox[0];\n    msg.y = box.bbox[1];\n    msg.width = box.bbox[2];\n    msg.height= box.bbox[3];\n    msg.class = box[\"class\"];\n    msg.score = box[\"score\"];\n    node.send(msg);\n}\n\n\n","outputs":1,"noerr":0,"x":1030,"y":210,"wires":[["bbd8edb4.45372","ac45a3a3.d45b5"]]},{"id":"bbd8edb4.45372","type":"debug","z":"2d489fd9.1eedd","name":"rectangle","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":1240,"y":210,"wires":[]},{"id":"fba73baf.e43ef8","type":"inject","z":"2d489fd9.1eedd","name":"test","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":130,"y":80,"wires":[["dfc95a0d.0305c8"]]},{"id":"99bb537b.b3b73","type":"change","z":"2d489fd9.1eedd","name":"pay to detect","rules":[{"t":"move","p":"payload","pt":"msg","to":"detect","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":840,"y":210,"wires":[["9031a37c.91ff"]]},{"id":"9a7b8548.3747c8","type":"function","z":"2d489fd9.1eedd","name":"","func":"var time = new Date().getTime()\nnode.warn(time)\n","outputs":0,"noerr":0,"x":570,"y":140,"wires":[]},{"id":"3bf60ae8.caf4f6","type":"mqtt in","z":"2d489fd9.1eedd","name":"","topic":"image/#","qos":"2","datatype":"auto","broker":"2a019090.5ba4d","x":160,"y":240,"wires":[["adb4835c.d0472","1804351d.75eb0b"]]},{"id":"adb4835c.d0472","type":"switch","z":"2d489fd9.1eedd","name":"","property":"payload","propertyType":"msg","rules":[{"t":"istype","v":"buffer","vt":"buffer"}],"checkall":"true","repair":false,"outputs":1,"x":370,"y":240,"wires":[["2085ac81.9563d4","9a7b8548.3747c8","2fccc750.eba658"]]},{"id":"1804351d.75eb0b","type":"debug","z":"2d489fd9.1eedd","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":360,"y":310,"wires":[]},{"id":"ac45a3a3.d45b5","type":"delay","z":"2d489fd9.1eedd","name":"","pauseType":"rate","timeout":"5","timeoutUnits":"seconds","rate":"1","nbRateUnits":"3","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"x":1090,"y":130,"wires":[["8a69170a.2a6d78"]]},{"id":"31b5112.2d60fee","type":"ui_group","z":"","name":"detection","tab":"da82a0d.14db76","order":2,"disp":true,"width":"18","collapse":true},{"id":"2a019090.5ba4d","type":"mqtt-broker","z":"","name":"","broker":"192.168.0.240","port":"1883","clientid":"","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","closeTopic":"","closePayload":"","willTopic":"","willQos":"0","willPayload":""},{"id":"da82a0d.14db76","type":"ui_tab","z":"","name":"Slide","icon":"dashboard","order":2,"disabled":false,"hidden":false}]
13

Loving what you guys are doing !

14

nice add !

15

Thank you for creating this node @dceejay !
Do you have any idea why your node is not working on WIN10 (see error log at the start of the post)? The bad luck ?

16

I know, this was for @dceejay but I could not keep my mouth shut; I believe it is because the javascrript version tensorflow (tfjs) is not supported on your platform. I got this error message on the Nvidia Jetson Nano when I tried installing

npm install @tensorflow/tfjs-node-gpu

GPU-linux-1.7.0.tar.gz
* Downloading libtensorflow
(node:9011) UnhandledPromiseRejectionWarning: Error: Unsupported system: gpu-linux-arm64
17

oh come on :wink:
yep, I saw that shows you a plug-in error on your platform, but me it says

"Error: Cannot find module ..."

Can we conclude that the platform is not compatible or do I have a module to add? so which one?

18

hmmm got the same thing trying to install it on a Pi Zero W (just thought I'd try) though mine ends with Error: Unsupported system: gpu-linux-arm

19

So for now, the only hardware that works is the RPI 3B (for me).
Surely the RPI 3B + ...

20

I have it working on a RPI 3B+