Just to get this straight - you are trying to authenticate your users before they are allowed access to the Node-RED Editor? Or is it access to a web page served by Node-RED?
In either case, trying to encrypt in the browser is going to be the wrong approach. Encryption is generally reversible and doing it in the client creates all manner of security vulnerabilities.
You can cryptographically hash values in the client because that can't be reversed and it can be useful to hash a password before sending it to the server.
However, you still need to use TLS to give wire-level encryption from client to server. As Colin says, that can either be done using Node-RED itself or (probably more scalable) using a reverse proxy to terminate the TLS.
Using something like NGINX, IIS, Apache or whatever also lets you add user authentication (and even authorisation) using the proxy. This is generally easier to manage securely than trying to squeeze everything into Node-RED (though it can do it as well if needed).
ALWAYS use standards-based authentication and authorisation methods - NEVER roll your own as you seem to be trying to do.
All of the above is friendly advice, not professional guidance.
Above all else, if you need security and you aren't sure how to do it - get a professional - at least get your work tested by professionals.
Security is hard to get right but very easy to mess up.