I've been playing with node-red for awhile now as a hobby but dreaming I could use it on my day job where we create RESTful json apis using node.js.
After looking pretty hard I've not found good solutions to:
a. All the flows in one giant json file (we will have hundreds of flows created by different teams where we have to restrict access who can access and edit which flows - having all that in one single giant json file is a non-starter)
b. Storing the flows in a database instead of the file system (I've studied the storage api and tried out e.g. node-red-flows-mongo but am I right the save part of the storage api assumes you are saving all the flows at once i.e. "a" above).
I know a lot has been happening re the editor/run-time split and the 1.0 beta release - is there anything with this work that would provide hooks I could plug-in to customize the above?
e.g. I wish the editor and storage api had something where the user would save their changes to the flow on a single tab and POST the json for just that flow to be saved.
The closest I see to that is saveLibraryEntry but I confess I've not tried to use the library features at all... is it conceivable I could save each of my tabs flows as individual "library entries" as database records, and then have my "getFlows" query those to dynamically aggregate back together the giant all-flows json?
Or can anybody think of anything better?
There is nothing new in this area in 1.0. the runtime/editor split work was in 0.20 and there hasn't been any new changes to that since.
The storage API gives you everything you need to save things however you want. Yes it's a single call to save your flows, but you can split up the json you given and store the different parts wherever and however you want.
There are no plans to change the editor to only deploy the current tab.
After 1.0 is done, there is scope to think more about how we can better accommodate multiple users trying edit the same set of flows at the same time. Just depends whether anyone interested in that is interested enough to help make it happen. Otherwise it sits on the backlog along with everything else and waits it's turn.
2 Likes
Thanks (and node-red is awesome btw my frustration is that it's soooo close to being exactly what my work project needs but with little things like the above being show stoppers).
I did think about splitting the big json up behind the saveFlows call - but I'd have no way of knowing what really changed in the big json right so I'd have no option but to split it up and rewrite everything on every save.
Or is there some meta-data with each of the tabs/flows e.g. modified by, modified datetime something like that? Or are there ways I could incorporate such "meta data" myself?
I should have said btw I am "interested enough to help make it happen" and I'm experienced with node.js and browser javascript it's just I'm new to understanding the node-red design (I've looked a bit at the node-red code but there's quite a bit to it!! 
)
2 Likes
I have the exact same requirement too.
I have worked for a long time in operational automation for large enterprises (and small ones too).
Current way of storing ALL flows in a single JSON is possibly the biggest holdback I have before I can push the awesome node-red for enterprise ready use.
I am also extremely interested in helping to make this happen.
Need some guidance on design of where the flows are getting stored and loaded from (where the json is actually read and written to) and how can we make it render via a DB query.
I already have a UI to manage Role based access to edit, execute, control a list of flows that are visible, executable, approval workflow for optional maker-checker functionality (where human in the loop is mandatory for compliance reasons).
Hi @abhutada
as I mentioned in my previous reply, you can use the Storage API to create a custom storage plugin for the flows.
https://nodered.org/docs/api/storage/
It seems to me that the use-cases in this thread are calling for multiple Node-RED instances, not so much for splitting up the flows. The NR process is not that heavy, running one per user or per application domain (whatever that ends up being) seems like the right solution.
There are access control issues mentioned in this thread that cannot be solved either, so that points to multiple instances. Even if all this was solved, the node runtime is still single threaded, so one slow function node holds everything else up, plus, I assume that in these use-cases you'd be running on a server with a large number of cores and then you'll be asking why does node only use one.
In other words, I do not believe " it's soooo close" is accurate in the sense that that statement was meant, there would be a ton of work necessary. But I believe it's actually already there: just run multiple instances.
1 Like
I tried the following 2 options.
The YAML solution (multi-flows) seems to be working closer to what I want - saving each flow in a separate YAML file.
However when using the postgres option, could not get too far:
What's passed to the storage API is just the same JSON object that is stored in the flat file when using the default storage plugin.
All the existing storage plugins are going to just store that object as that is what needs to be returned to Node-RED when it requests the flow to run.
If you have different requirements (e.g. to store the tabs separately) then it is going to be up to you to implement that by understanding the structure of the flows JSON object so you can break it up and rebuild it from the parts later.
Agreed. However, the node-red-contrib-multi-flows-storage solution seems to be already splitting each flow into a separate JSON / YAML (based on what you choose).
If each tab JSON is already available, is there a way to selectively Deploy only a specific tab (from its corresponding JSON) into another NR environment programmatically?
