We are trying to use Node-RED as a rock solid solution for Integration of our micro services ecosystem, we have also achieved some flows on our development environments till now, but we want to make sure we are going the right way since we are facing lot of roadblocks in achieving the final vision for our team and the solutions we are trying to build upon them.
Some more Info about the way we have deployed and using Node-RED-
We have 2 instances of Node-RED(one for creating and testing flows and other is just read-only) deployed over our Kubernetes Cluster
We are monitoring the performance of Node-RED pods using Prometheus & Grafana
We are working on establishing SSO for Node-RED using our cloud gateway
We are using the Node-RED project functionality to seamlessly integrate with our git
We have used adminAuth to ensure that different users login using their ldap user id and password and while pushing the flows to git repo, their unique credentials are used
We have using Jaeger for its tracing requirements
Vision we are trying to achieve:
We are in process of creating core nodes for our business logics or Lambdas, micro services, messaging queues, etc
We want to have a platform where users with very less idea on programming/coding can use these core nodes to orchestrate their environment and services
And then can deploy these flows to whichever environment required through our CI/CD pipeline
Now the question comes where is the problem and why even I am writing this post here ?
We all know that Node-RED is not multi-user and there are no standard approach or plugins/hooks/wrappers to achieve the same
The above point might become a big roadblock for us, since we cannot assume that for the environment which any team will be integrating will be a single person team, so collaboration
stands out as a big question here
When multiple devs are working in our team on the same instance of dashboard and when we push and pull to our repo, while creating Merge Requests many of times we are facing merge conflicts (which is a common thing to have)
In order to merge the process of resolving conflicts is getting too complex because of a single flows.json file
So the next big challenge which we see is having all flows in a single file, which is becoming hard to manage from versioning and git perspective
Also when trying to resolve merge conflicts on git, its very tough to understand the diffs because of different changes which are coming from other devs
While resolving conflicts many a time the json diffs for wires and x,y coordinates are very confusing if you want to ensure you are doing right thing
Current situation: We are not sure if we have picked a correct tool for achieving our business use-case
May be we are trying to achieve something for which Node-RED was never designed for and due to which we are spending much time in achieving something other than our actual business use case
Posting all this here with all due respect to Node-RED and the complete team behind this.
1/2 - personally, I would look at implementing authentication and authorisation outside of Node-RED if you can. I've no doubt that Node-RED will catch up but it is currently fairly simple on that front. As you already have a proxy, perhaps see if you can do what you need there instead.
For the rest, you are somewhat ahead of the leading edge here I suspect. We know that, now v1 has been released, designs for flow files may mature to allow more complex scenarios such as the ones you describe but for now I'm not sure there is much that can be done without writing some "interesting" code outside of NR to allow split and rejoin of flow files (and credentials which will be even worse as they are encrypted).
Alternatively, accept the overhead of having multiple Node-RED instances which can easily scale using various methods but certainly requires additional resources.
Or maybe a hybrid with multiple dev instances with some clever script along with development standards to merge flows.
Improving the multi-user collaboration experience is certainly something we want to do. It's just a case of prioritising that work against everything else.
Given you want to use the projects feature, then the approach of writing a custom storage plugin may not be feasible - as you'd lose the projects functionality unless you copied all that code as well.
Providing an option to split the flows into separate files is definitely a feature I'd like to see us add.
You'll still have issues when two users modify the same tab - at some level, merge conflicts are going to be hard to avoid and some effort has going into the merge conflict resolution tooling in the editor itself. But in absence of any particular feedback on it, there aren't any specific plans around it.
I am working with @gauravdasgupta on this Node-Red integration.
Concerning "You'll still have issues when two users modify the same tab" I would propose to lock the modified flow and display information about who is working currently on the flow to the other users.
the single read-only executor Node-Red instance can be scaled horizontally in case of the heavy load. In this case, I see a need of an additional db-kind contextStorage (besides existing memoryOnly and file), which can be shared between multiple Node-RED instances;
we are expecting to have 200+ different core flows + customers flow. In this scenario, it is quite hard to work with a flat list of flows when you can see only 5-6 flow names as tabs. As an experiment, we have tried to integrate a node-red in an iframe with some Tree component in the left menu. API to work with the tree model was exposed from node.js which was hosting Node-RED as middleware. Tree structure was stored in an additional json file to keep the original flows.json untouched. See the screenshot:
Working this way is more comfortable. And of cause, it would be nice to have this tree support as a part of the standard package.
Proposals:
Current state (single flows.json): add a grouping possibility inside an existing object model; possible future state (separate flow files): store an additional index of flows with a tree structure;
Replace the current left menu with list of nodes to the tree structure of the flows;
Tree component on the left has to support working with groups and rearrange flows with drag-and-drop. For our POC we have used this component: Angular Tree Component;
Display list of the placeable nodes by clicking existing PLUS button (which currently creates new flow);
our tree view supports grouping related flows with multiple level of hierarchy and move flow from one group to another (drag-and-drop) with all expand collapse on the group of cause.
