I embedded node-red in an express app as to allow me to start the app with pm2 in cluster mode.
I start two different express apps with node-red embedded. My architecture is based on a microservice architecture, thus I have a node-red instance running on different ports offering different functionality.
I use mqtt as the "inter node-red instance" communication layer.
In my current scenario my first node-red instance is an http service. It receives http requests then via mqtt sends it to my second node-red instance that performs business logic. The second instance will then respond back to the http service via mqtt and the http service will then respond back to the client.
I have spoken about this sort of architecture before and the group managed to assist me in the issues I had with keeping track of the http req and res, so as to allow me to always respond back to the correct client. I use this by storing the http req and res objects in a global context and then cleaning it up after a response was successful.
I have now gone one step further, where I use pm2 to start more than one instance of my express apps. With pm2 you can configure the startup process to use cluster mode and it will then start as many instances of your nodejs app as you have CPU's available. This actually works very well and there is no problem with how it operates.
I have actually run into a problem with my global context now. Let's say that I have two CPU's then two instances of the embedded node-red application will start. Under the hood, this uses the Node.js cluster module such that the scaled application’s child processes can automatically share server ports.
If I save my http req and res in a global context then there is no guarantee that the same instance of node red will process the response, thus there will be no context set for the request/response. This is a hit and miss scenario.
I was thinking to use Redis as an in memory data store where I can push and pop the required data, but this unfortunately does not work, as I am unable to store a http res in Redis.
When trying to store it as an object, it always stores it as [object Object], which when you pop it from Redis, it is just a string which means nothing. If I try to stringify it, get TypeError: Converting circular structure to JSON and it is not pushed into Redis.
If it is obvious mistake I am making, I would appreciate your assistance or if there is another way of doing this, I would really appreciate your help.