My team(@shrayank and @m0nirul) is currently working on integrating Node-RED in an already built IOT platform and had a couple of questions:
Our platform works on multiple instances and we would like to implement a functionality where we can run each flow separately in its own instance or to be more specific its own container. This would require us to override the basic functionality of starting and stopping a flow and also the message routing to basically communicate between instances.
Is this in any way possible to achieve without overriding any deep functions like emit as we would like to upgrade node-RED when an update is available without too many changes in the codebase. I saw the pluggable message router page but was unable to find anything definitive in implementation or the progress.
An exposed logger to interact with or use the logs generated by node-RED in the parent application.
We would be open to work on a pull request for this if needed.
The scenario you describe is one we are slowly working towards enabling to various degrees - but we aren't there yet.
The 0.20 release split the runtime and editor components of the internals - that was a big step as it allows custom code to be put on the path between editor and runtime. It could allow, for example, the code to intercept the requests to deploy and get the active flow configuration and perform whatever additional work you want to split the flow up and distribute it. That's the theory - but no-one has done that to my knowledge.
As we described in our recent update to the roadmap, the pluggable message routing feature is not currently in the plan for version 1.0.
As I can see from the version 0.20 release, there are two possibilities to support multiple instances:
Intercept the requests between editor and runtime as you mentioned:
As I can see right now, editor-api uses runtime to initialize. Therefore if I would want to enable multiple runtimes, intercepting these requests and distributing them is not easily done as these are now tightly coupled. We will have to initialize the editor with a middle layer which can then distribute these requests, and would require changes on the runtime side to accommodate the new communications protocol.
Intercept the request between editor-client and editor-api:
If we were to use a loadbalancer capable of using reverse proxy like HAProxy or even our own implemented loadbalancer with a runtime in memory DB for flow sessions it would be possible to scale the runtime. This would also require the start of the client independent of the editor-api.
For inter-instance communication socker.io can be extended to use a in-memory DB like redis/a message queue like kafka and emits can be transmitted across instances.
What could be the better architecture, if we were to move forward. Or is there something better I missed.
I don't see anything that would be a PR here, unless I'm missing something you are proposing. There are lots of possible ways to build the sort of thing you are wanting to do. The reality is 0.20 has only just shipped so I can't claim we have a great depth of knowledge about what the best way to make use of it is. There are options. You need to evaluate those options for your own use cases.
The PR I am proposing is to help enable multiple instances on Node-RED easily.
First is to decouple the editor UI and editor API completely. Now editor UI is being started from editor API. I am proposing to move this initialization into the main red.js. This will help us to start the editor UI without editor API which can in turn be deployed in separate instances. This can help in easily introducing a middle loadbalancing/cluster balancing layer.
Also to help multiple instances communicate, initializing socker.io with a DB in case of multiple instances are being used. Socket.io can be passed as an argument when initializing Node-RED in embedded mode.
Second would be the implementation of the load balancing/cluster managing middle layer. This would be a core implementation if support for multiple instances is offered fully with the core.
