Usage
Takes a buffered image and runs TensorFlow Facial Detection and/or Facial Recognition to Detect:
- Faces in an image
- Facial landmarks
- Compute Face Descriptors
- Face Expressions
- Age & Gender Recognition
- Face Recognition by Matching Descriptors
13 Likes
Hi @meeki007,
Thanks a LOT for this contribution! Face recognition has always fascinated me, but it was way too much out of my comfort zone... Hopefully your nodes can make it accessible for noobs like me.
I have not tried your node, since I'm in the car now waiting until the swimming class of my son is finished. So my questions are based only on what I read now 
Please forgive me if I have misinterpreted your nice documentation on my small cold smartphone screen!!
Am I correct that it works like this:
- I put some close-up facial pictures in a well defined folder structure.
- Every time images are added or removed, I need to send a message with
ReInitializeFaceMatcherto the FaceRecognition. - The training takes some time, and then I get an output message (so my flow knows it can start sending images). And meanwhile you simply ignore any image that arrives via an input message.
- The training result will be stored in the flow memory context. So when I have persistent context, the training result will be reused after a restart (which allows me to avoid having to repeat the training every time).
- When an image arrives via an input message, the face recognizer will send as output and array of names (each with their probability percentage).
- We can specify in the config screen (or input message) what to do with the result: send an output message always, or don't send an output message when the probability is below N percent.
- There is an option to show or hide the border (with the name and perhaps percentage) around the face.
Is that correct? Or did I just send a large feature request list, which turns me suddenly into the most greedy member of this community 
Awesome job!!!!
Looking forward to experiment with it,
Bart
3 Likes
#1 Correct !
#2 Correct-ish you don't need to send a msg.settings its optional use. So you can ReInitialize via a input msg. The other way is you just redeploy the node/flow that works too!
#3 False Some time for 20 images (4 diff people) its about 6sec. I think your confused by the image you send to start training. The image you send is the one your want to ID / get info about. When you send this image and the node needs to ReInitializeFaceMatcher because you have added people/faces to the folder it just adds the extra 6sec to load the new people/faces. Then It takes the photo you sent and checks it to see if its a match. After that all new images you send to the node via msg do not require the 6sec load because its been saved as a context for use on any image you send. times should drop to milliseconds for all new images you send now.
#4. False its not training. the face marcher descriptor(s) values are stored in context. Do not restart. any deploy and these values are lost from context. I guess this is what you could call a possible new feature. persistant storage of above. I did not do this because I wanted a easy way for users to reset this context when they add or remove folders/faces.
#5 Correct-ish it will output a object. in this object you will find a object array with the name results. in this array you will have objects with allot of information based on the detentions you set.
#6 half correct there will always be an output msg. based on the settings you select. if you set your N percent too high you may lack a result but there will always be information in the output msg for you.
#7 yes but the border and names are not from my node. It is from the optional nodes you can add.
node-red-node-annotate-image
node-red-contrib-image-output
I created a function to format the output data from my node to send it on to these nodes. yes you can change the box size name size color etc.
Not at all greedy. People have good ideas I never think of. If I like them I'll peck and add them. If im neutral about the idea I say things like "code it up and send me a pull request on get hub so I can merge it"
If i don't think it fits in my node I'll say no and explain why.
Many times im chasing the might dollar and dont have time to fix things and I say so.
I found it hard to stuff every thing involved in facial recognition into a single node. I've played with object detection in nodeJS for a year now and wanted this to be as user friendly as possible. But OH the documentation on this is going to need allot of work. Thanks for your questions. I have a few ways I may change some of the documentation tomorrow to help others.
If you dont want people asking you too many questions FIX YOUR DOCUMENTATION
Best of luck,
Meeki007
1 Like
NOTE to all I have yet to run this via GPU so if you give GPU a go and there is an issue please let me know. I have a jetson-nano on my to buy list Jan 1st. Ill be able to test it then.
Thanks for your time!
Yes I think I'm messing up things in my head. I had always thought images were used for 3 purposes:
- That you need a large dataset/database of face images, to train the library to recognise faces in general. Which takes a long time and uses lots of system resources. But that is perhaps not required?
- Then a series of face images from known people that you want to recognize. So I assume it is this that takes 6 seconds. I thought this was also called "training"... But am I correct that you call this "face marching" then?
- The image that you want to recognize
Just an idea: can't you (if you haven't done it already) show a dropdown of possible storage locations on the config screen? For example if I specify that the descriptors need to be stored in "flow.my_descriptors" and I tell Node-RED that my flow context should be persisted to disc, that it will survive any (re)deploys or reboots? And that this flow variable content is updated every time that it is required.
But perhaps I'm misinterpreting this as a face recognition noob...
And I also use the coco-ssd nodes. Do you know whether this is a single tensorflow instance that serves both your node, and the coco-ssd node? Because I read here (under "prerequisites") that there might be conflicts if you multiple (offline) instânces of tensorflow running. But that perhaps has nothing to do with this ...
Bart
training has been done already. I load them in for you.
faceapi.nets.ageGenderNet.loadFromDisk(modelPath),
faceapi.nets.faceExpressionNet.loadFromDisk(modelPath),
faceapi.nets.faceLandmark68Net.loadFromDisk(modelPath),
faceapi.nets.faceLandmark68TinyNet.loadFromDisk(modelPath),
faceapi.nets.faceRecognitionNet.loadFromDisk(modelPath),
faceapi.nets.ssdMobilenetv1.loadFromDisk(modelPath),
faceapi.nets.tinyFaceDetector.loadFromDisk(modelPath)
Correct
This is a hard one. I need to initialize it.
const faceMatcher = new faceapi.FaceMatcher(labeledFaceDescriptors, this.Face_Recognition_distanceThreshold);
nodeContext.set('faceMatcher',faceMatcher);
nodeContext.set('FaceMatcherInitialized',true);
//msg.faceMatcher = faceMatcher;
}
var faceMatcher = nodeContext.get('faceMatcher');
then I call on it to do the matching
//sort detections single/multiple faces
if ( this.Tasks === 'detectAllFaces' )
{
result = result.map(fd => {
const the_object = fd;
//add the match to the object
the_object.match = faceMatcher.findBestMatch(fd.descriptor);
return the_object;
});
}
Ill think about how I can do persistence.
Also I kept it in context to each individual node so a user can have multiple different folders of faces for different detector needs.
I'm liking the thought process. It would be cool to not have to load it all every time you redeploy a node. The question is not if I can do it. I'm just thinking about how im going to do it in a way that is good for users and good for the node
That makes sense!
There are already other discussions about that. I have always taken as rule of tumb that I don't know if there is even a filesystem available, so my nodes are not responsible for persisting data. I add it to context, and the user decides whether the context should be persisted. But now I'm going a bit off topic, and taking away the focus from face recognition itself...
1 Like
No you just solved the problem!
I read that and the solution clicked in my brain.
Output the value of
const faceMatcher
to the output payload object.
then if the user wants to store the data he/she can.
then make a input for that data in the msg.options input and they can load it in as an object.
and then I can load it in
the_object.match = faceMatcher.findBestMatch
Well that will be on Sunday for sure I'll poke at that.
Hmmm that would indeed be a way to do it. But then I would have to add an extra node to store it. And after a deploy I have to read the value in my flow and put a wire to load it again in your node.
I like to keep my flows as simple as possible, so would be a pitty if your node wouldn't be able to able to store its own context... But of course it is your node, so you can decide. If you want, I can try to create a pull request tonight. Not that you need to merge it, but just to have something to discuss about...
I just started playing with the node and added a couple named_folders in with some images and ran it to do facial recognition. With the first test it worked fine and I added another folder and some more images and ran. The node sat there with the blue dot for ever. I finally looked at the node red log and see this:
12 Dec 07:32:44 - [info] Started flows
(node:18404) UnhandledPromiseRejectionWarning: Error: Size(1048576) must match the product of shape 512,512,3
at inferFromImplicitShape (/Users/Paul/.node-red/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:884:19)
at forward (/Users/Paul/.node-red/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:7427:17)
at /Users/Paul/.node-red/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:3480:55
at /Users/Paul/.node-red/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:3319:22
at Engine.scopedRun (/Users/Paul/.node-red/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:3329:23)
at Engine.tidy (/Users/Paul/.node-red/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:3318:21)
at kernelFunc (/Users/Paul/.node-red/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:3480:29)
at /Users/Paul/.node-red/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:3492:27
at Engine.scopedRun (/Users/Paul/.node-red/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:3329:23)
at Engine.runKernelFunc (/Users/Paul/.node-red/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:3490:14)
at reshape_ (/Users/Paul/.node-red/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:7432:19)
at reshape__op (/Users/Paul/.node-red/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:4258:29)
at Tensor.as3D (/Users/Paul/.node-red/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:29472:12)
at /Users/Paul/.node-red/node_modules/@vladmandic/face-api/dist/face-api.node.js:8:34933
at Array.map ()
at /Users/Paul/.node-red/node_modules/@vladmandic/face-api/dist/face-api.node.js:8:34594
at /Users/Paul/.node-red/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:3319:22
at Engine.scopedRun (/Users/Paul/.node-red/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:3329:23)
at Engine.tidy (/Users/Paul/.node-red/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:3318:21)
at Object.tidy (/Users/Paul/.node-red/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:8898:19)
at NetInput.toBatchTensor (/Users/Paul/.node-red/node_modules/@vladmandic/face-api/dist/face-api.node.js:8:34541)
at /Users/Paul/.node-red/node_modules/@vladmandic/face-api/dist/face-api.node.js:8:100674
(node:18404) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:18404) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
This was using the first example flow with face recognition turned on.
OS: macOS - v10.14.6
Node-Red - v1.2.5
node.js - v12.13.0
Let me know if there is anything else you need.
Check that you sent it a image. prefer jpg. I think you have a funky img file in that folder
I do have a png file, didn’t know that might cause an issue. I’ll remove it and report back
I keep getting that dump in the log. I did have a PNG file but I removed it. What I'm now using is a HEIC photo, taken on my iPhone, that I convert to jpg. This is the latest in the log:
Welcome to Node-RED
12 Dec 13:49:44 - [info] Node-RED version: v1.2.5
12 Dec 13:49:44 - [info] Node.js version: v12.13.0
12 Dec 13:49:44 - [info] Darwin 18.7.0 x64 LE
12 Dec 13:49:44 - [info] Loading palette nodes
node-pre-gyp info This Node instance does not support builds for N-API version 6
node-pre-gyp info This Node instance does not support builds for N-API version 6
2020-12-12 13:49:46.357584: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x105c5ded0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-12-12 13:49:46.357634: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
12 Dec 13:49:47 - [info] Worldmap version 2.6.0
12 Dec 13:49:48 - [info] Dashboard version 2.24.2 started at /ui
12 Dec 13:49:48 - [warn] rpi-gpio : Raspberry Pi specific node set inactive
(node:29226) [DEP0016] DeprecationWarning: 'GLOBAL' is deprecated, use 'global'
12 Dec 13:49:48 - [info] Settings file : /Users/Paul/.node-red/settings.js
12 Dec 13:49:48 - [info] Context store : 'memory' [module=memory]
12 Dec 13:49:48 - [info] Context store : 'default' [module=localfilesystem]
12 Dec 13:49:48 - [info] User directory : /Users/Paul/.node-red
12 Dec 13:49:48 - [info] Server now running at http://127.0.0.1:1880/
12 Dec 13:49:49 - [info] Active project : Facial_recognition
12 Dec 13:49:49 - [info] Flows file : /Users/Paul/.node-red/projects/Facial_recognition/flow.json
12 Dec 13:49:49 - [info] Starting flows
12 Dec 13:49:49 - [info] Started flows
(node:29226) UnhandledPromiseRejectionWarning: Error: Expected image (BMP, JPEG, PNG, or GIF), but got unsupported image type
at getImageType (/Users/Paul/.node-red/node_modules/@tensorflow/tfjs-node/dist/image.js:351:15)
at Object.decodeImage (/Users/Paul/.node-red/node_modules/@tensorflow/tfjs-node/dist/image.js:196:21)
at image (/Users/Paul/.node-red/node_modules/node-red-contrib-facial-recognition/facial-recognition.js:301:33)
at /Users/Paul/.node-red/node_modules/node-red-contrib-facial-recognition/facial-recognition.js:372:50
at async Promise.all (index 0)
at async facialrecognitionNode._inputCallback (/Users/Paul/.node-red/node_modules/node-red-contrib-facial-recognition/facial-recognition.js:799:46)
(node:29226) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
(node:29226) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
So worked with my photos. Not working with your photos. I deal with jpg only. I was going to put a image checker in there but it adds time. Ill take a deeper dive Sunday but with out a copy of what your using to make it fail im at a loss.
If I user your fourth example (from the GitHub page)
and set 'FaceRecognition' to 'disabled' the flow runs fine.
If I set it to 'enabled' it sems to wait forever but looking in the log I see it died with the following:
12 Dec 16:51:52 - [info] Started modified flows
(node:35531) UnhandledPromiseRejectionWarning: Error: Expected image (BMP, JPEG, PNG, or GIF), but got unsupported image type
at getImageType (/Users/Paul/.node-red/node_modules/@tensorflow/tfjs-node/dist/image.js:351:15)
at Object.decodeImage (/Users/Paul/.node-red/node_modules/@tensorflow/tfjs-node/dist/image.js:196:21)
at image (/Users/Paul/.node-red/node_modules/node-red-contrib-facial-recognition/facial-recognition.js:301:33)
at /Users/Paul/.node-red/node_modules/node-red-contrib-facial-recognition/facial-recognition.js:372:50
at async Promise.all (index 0)
at async facialrecognitionNode._inputCallback (/Users/Paul/.node-red/node_modules/node-red-contrib-facial-recognition/facial-recognition.js:799:46)
(node:35531) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 4)
This is with the images provided with the node.
1 Like
I fixed it while you were trying to help me with more information.
I just never expected users to send bad files. It's not your fault that the node does not think your image is valid. I just never sent it a invalid image when i was testing the node. This is a good early bug to find.
I've created catches for this issue.
If a user puts a file that is bad (in the eyes of tensorflow) it will let them know now and tell them what file it is.
if a user sends a bad buffered image it will now tell them
thanks for working with me on this @zenofmud
All known UnhandledPromiseRejectionWarning's are now handled. I'm sure people will help me by finding new Unhandled ones 
3 Likes
Give me 15min to push an updated node.
Update to this msg ------ > I almost pushed it with the badfile.jpg i created for testing ...... slow down there buddy...
1 Like
update pushed to npm and github. Updated with node-red. version 0.25.89
you may need to restart node-red so it will show up in the pallet manager.
you can view my changes here ---> https://github.com/meeki007/node-red-contrib-facial-recognition/pull/1/files
If my fixes don't work I did notice some windows and mac stuff @ https://www.npmjs.com/package/@tensorflow/tfjs-node
However because it does work (most the time) before you started adding your own images I don't think this below applies to you.
Windows / Mac OS X Requires Python 2.7
Windows & OSX build support for node-gyp requires Python 2.7. Be sure to have this version before installing @tensorflow/tfjs-node or @tensorflow/tfjs-node-gpu . Machines with Python 3.x will not install the bindings properly.
For more troubleshooting on Windows, check out WINDOWS_TROUBLESHOOTING.md.
Mac OS X Requires Xcode
If you do not have Xcode setup on your machine, please run the following commands:
$ xcode-select --install
This (the above) should not matter for you but a dependency that went unnoticed could be having an effect on images. Do not own a MAC or have a single windows PC in my possession
