Share flows between instances

[Dunken]

I have independent Node-RED instances deployed and running. I would like to add something like a library to share flows among NR instances.

Can anyone help me get started with plugging in my own filesystem (e.g. S3 or EFS)? The instances should not be able to overwrite each other's library elements (flows) but to create new ones.

Important: the sharing of any flow should be optional. This means that the default location of a Node-RED instance must only be accessible by the owning instance.

[zenofmud]

Currently there is the ability to save flows using the export option othe hamburger tab (top right)

One 'idea' is that you create a folder in .node-red/lib/flows called 'shareFlow' so you end up with .node-red/lib/flows/shareFlow. Then, in a seperate tab on each NR instance, you have a flow that contains a watch node that would watch the .node-red/lib/flows/shareFlow folder and when a new file appears, it would write it to an actual shared folder somewhere on your on your system.

You would have another flow that watches that folder and when a new entry arrives, it would copy it to that devices .node-red/lib/flows/shareFlow and it would now appear in the import options for that version of Node-RED.

This is an untested idea and others might (probably) come up with something better but it gives you a way to start experimenting.

I'll be looking forward to reports of your progress.

[Dunken]

Thanks for the inputs! I would like to avoid any business logic in this regard in any flow, that's why I would deploy the file-watchers rather in the backend directly.

In addition to the general approach I also have been looking for how to best inject a new storage layer (S3, EFS or even a database).

[TotallyInformation]

@zenofmud

One 'idea' is that you create a folder in .node-red/lib/flows called 'shareFlow' so you end up with .node-red/lib/flows/shareFlow . Then, in a seperate tab on each NR instance, you have a flow that contains a watch node that would watch the .node-red/lib/flows/shareFlow folder and when a new file appears, it would write it to an actual shared folder somewhere on your on your system.

If all the instances are on the same device or use a shared filing system, you could simple use filing system links rather than having to mess with some kind of watch process.

@Dunken

In addition to the general approach I also have been looking for how to best inject a new storage layer (S3, EFS or even a database).

For flows? Not sure that the pluggable flow system is ready for use yet is it? I know that some events have been added in preparation.

[Dunken]

@TotallyInformation

For flows? Not sure that the pluggable flow system is ready for use yet is it? I know that some events have been added in preparation.

Yes, for flows. I'm wondering if I have to use storageModule

[TotallyInformation]

@Dunken

Yes, for flows. I'm wondering if I have to use storageModule

Ah, yes, that's it. I'd forgotten that was there.

[knolleary]

When you say you want to share flows between instances, if you mean you want two instances of Node-RED to load the same flows, then you can use a custom storage plugin as others have pointed you at.
For example, the IBM Cloud instance uses a Cloudant based storage layer for storing flows etc. node-red-app/cloudantStorage.js at master · IBM/node-red-app · GitHub

If by sharing you mean you'd like two separate instances of Node-RED to easily share flows and allow the users to import/export flows that are shared, you can use the pluggable library system that was more recently introduced. This will show up in the import/export dialog as a new library source - alongside the 'local' and 'examples' libraries that are there.

Details of that plugin api are here: designs/pluggable-library.md at master · node-red/designs · GitHub

[Dunken]

That's a good question... I guess I tried to solve the latter use-case with a storage plugin...

It's definitively separate instances. I also believe the assets should show up in the import/export dialog. Is it now a new source or is it part of 'local' I don't care too much to be honest (if it's under 'local' I would have imagined something like a 'mySharedAssets' folder (writable - will be shared with others) and a 'readSharedAssets' folder (read-only - contains assets of other NR instances 'mySharedAssets' folder).

I definitively don't need an Editor plugin as I would configure the plugin during the deployment. However, just from looking at the referenced example it's unclear to me how I would have to to expose the different libraries... (e.g. does this Admin API now contain something like a /library endpoint?)

[knolleary]

1. if you want to do everything under the 'local' library, then that is part of the core Storage API.

2. If you want to add custom library sources beyond the 'local' entry, then yes, you need a library plugin.

The design note I linked you to was possibly a bit lower level detail than what you need.

A better example is the Library File Store plugin here: GitHub - node-red/node-red-library-file-store

That plugin can be used to add additional library sources that are backed by the local file system.

If you wanted the library to be backed by some other storage, then you'd need to create a plugin that does the work to talk to that storage. The file store plugin is the only one that exists, as far as I know, but the concept was always to create plugins for other common types of storage.