Subflow Synchronization

General
1

I run different projects on different servers. Many of them have common subflows. If I make an upgrade to a subflow, I manually change the version, and then export/import in ~15 other Node-RED instances.

Sometimes I don't do this process everywhere, and I always check on the manually set version, to see if there's a newer version somewhere. Sometimes I "upgrade" a subflow in one server, but forget about it, and upgrade a lower version on a different server, creating a complicated to investigate and solve desync.

This obviously got a bit out of hand in time and today I decided to ask about Subflow Synchronization.

I've noticed the question was asked in 2019, but maybe meanwhile there is a way.

If there isn't a way to do this yet, I'll create some hacky way and share it here.

2

The sensible thing to do since you already have a network of node-red instances would be to add another instance and use a flow instead of a subflow. Connected via websockets or some other connection such as tcp, udp or http. In other words convert the sub-flow to an API service.

3

Please don't take my counter-argument as a rude rejection of solutions, I intend it only as a polite discussion on the topic.

I do use this method in some cases but it creates a lot of network pressure on the "main" server and causes lag on very heavy use situations.

For example:

operagx77f3Ed9s
operagx77f3Ed9s669Ă—207 18.1 KB

Everything there except for the marked nodes is a subflow. On every step it would switch to another server and back.

An average msg is ~10kb.
There are an average of ~30 messages per second travelling through a server.

So this is a nice solution but, with some exceptions, I don't want to server hop. Different projects must be (in my case) mostly independent without a single point of failure. Lag on the main server means lag on everything.

4

Also most of my subflows rely on local global or environment values which can not be checked on a different server.

5

Made a python script which does the following:

  • pull latest from a shared project (using github read deploy keys setup once on each instance)
  • from latest, compare all subflows with current project, can be as easy as comparing versions
  • if newer versions are found, stop node red, delete subflow and nodes it contains from flow.json
  • insert subflow and nodes from shared project flows.json to current project flows.json
  • start node red

It also does some git commit/push before and after, and abort + revert if anything goes wrong mid process. But it works fairly smooth, no hickups so far. Luckily the flows.json is easy to work with raw and no surprises so far. Just be cautious that nodes inside a subflow are not inside the json object, but parallel. They are linked by subflow id.

This means whenever I update some subflow, I MUST apply it to the shared project, which is a bit of a pain to remember also, but at least distribution is easier. And I still have to run update script individually on each machine, but that also allows for manual checking everything works afterwards.

Additionally, to make it easier to log, I renamed all subflow ids (UUID) to human readable title like "subflow-title-of-subflow", for example "subflow-http-retry". If you choose to do this, just make damn sure you don't have naming conflicts :winking_face_with_tongue:

6

I know that some (many ?) are not big fan of using docker to run node-red. But....
Why not mixing the approach suggested by Julian (the API call) and deploy on each and every host a docker container containing the subflows ? You don't overload one central system. You avoid a single point of failure.
This can be easily automated with ansible, for example.

7

Ironically, I am one who does use docker. :laughing:

My structure is something like this:

VPS 1 (strongest too)

  • Non-Docker "Main" Project, also a main Database server

(all the rest bellow are Docker containers)

VPS 2

  • 8 separate independent projects for different clients

VPS 3

  • Some helper "separate processing server" for some Main Server tasks
  • Another sub-project of the main one
  • some other personal projects, some with websites

The is a very nice idea, of a "subflows docker instance", used by all the servers on that VPS. This would, in my case, turn updating lots of projects into updating just 3.

And now that I re-read your message, I think you're suggesting mixing this with @ThingsTinkerer's idea and then the 3 "subflow keepers" would sync themselves.

operaMiTcyeepVB
Seeing this, I think it's understandable why I was looking for an easy sync solution, in case it would exist meanwhile. It would take some time to switch everything to an API method, but maybe not more than a few worth-it days.

Thank you for the contribution, your idea is very nice. This topic will be seen by others too in the future, definitely a good idea for others if maybe not for me (I'll consider it).

8

You really solved the problem on your side. :star:

I thought about some flows.json manipulation too, but I thought of an "updater" non-docker Node-RED which would manipulate the flows in every docker folder on the VPS (because I have no Python experience). And then restart those servers. But it would cause some downtime on every update. I also did not investigate enough how difficult it would be to detect and change what exactly should be changed in the file. Your insight helps with that.

I also investigated the possibility to HTTP remote push changes to other Node-RED instances as an automated non-human-user account. Some kind of remote import and "deploy modified nodes only". But I got distracted with other things, I am not currently sure if that is possible. I remember it requiring a session token, at least.

9

Yes, but not using the same logic and not using NodeRed API.
If you manage the sunflower in a project. You could use GitHub actions and create the new container like that. Then you run the pull from vps , eg with ansible and you update the container on your systems.
Additional bonus in your situation, if you put your X clients in different docker bridge network, you could deploy the subflow container X time , one per bridge network. Like that the solution is quite scalable.

10

My solution to the problem of reusability has been to create a node package that commits my flows to a github repository. The node package also provides features for comparing local copies to server copies, i.e. in the git repository. I.e. if I have a common flow that I'm using across multiple NR instances, the workflow becomes:

  1. install flow code from github into an NR instance
  2. make changes to the flow as necessary
  3. compare the flow to the github version:
    a. if there are changes have been made by another NR instance, then I have to integrate those before pushing my new changes
    b. if no changes on the remote version, push changes to github
  4. on a second instance of NR, I can then just update the flow with those new changes or incorporate them to existing local changes

The github repository is the central, single source of truth and the node package is the coordinator in the local instances of NR, i.e., the node package is responsible for (visually) comparing, pushing and pulling of flow code.

I got that sick and tired of doing a export and import of flow code between my NR instances that I put this all together. You can try it out using the FlowHub node package. Works well enough for me, I maintain hundreds of flows using that setup, including the flow that maintains the node package.

11

Quite honestly, I would take a step back and see if you can't refactor some things. Could you, for example, rewrite your sub-flows as an actual node.js module? Seems to me as though this would be an optimisation step that would be worthwhile. As a module, this would easily be made available to all your instances and would better handle the throughput anyway.

12

Seems like a huge step in complexity? What if the subflow uses some other node like modbus or bacnet, it's easy to import those in node red, but how is it to import those in a separate js module? What about debugging, if a subflow doesn't behave, you can go into it and add debug nodes. In a js module, you must edit source code to add console logs, then make release and re-install? It's a long detour and high cost IMO. It solves the distribution problem, at the cost of sacrificing the ease of use of NR.

13

Not necessarily. Depends on the logic encapsulated by your sub-flow - which I've not looked at I'm afraid.

Yes, that might make things a little harder. You would need to do some additional configuration for sure. However, once done, it should be easy to scale.

Everything depends on the value of your customers, the complexity and your expected growth/expansion. You would need to review what is worth doing.

However, it would be a lot more robust and scalable.

Just because you have a hammer, not everything becomes a nail.

Node-RED is great. But not everything fits into it neatly. Nor should it.

That is pretty trivial. Node.js is just server-side JavaScript after all.

In addition, there are several ways to use modules with node-red. A Node-RED plugin that references your external module might be ideal. That way, you run the code inside of node-red and would have access to some things like global and flow variables and would allow output to Node-RED's logs and debug panel. A custom node might be even better as that would also potentially give you access to the msg data as well (as an in-line node) and would directly replace your sub-flow.

A custom node would use your external module which you would publish as a public or private npm package or reference direct from any other source available to npm: A local version or GitHub, ... - meaning that you can easily maintain a single instance of your source either privately or publicly depending on need but still access it from wherever you need it.

Yes, but that is easily handled with a simple, localised development and test environment. With multiple (presumably paying) customers, you should have that anyway. Node.js/npm handles this very well.

"long" and "high" are relative terms. As I say, only you can make commercial decisions about cost vs value. All I can say is that it is likely to be robust and highly scalable. Indeed, it might lead to other optimisations that you might either want or even need for the future. Or it might not. :smiley:

And it does not necessarily reduce the "ease of use of NR" - indeed, it could easily be the opposite by removing visual complexity from repetitive workflows whiles increasing throughput and scalability.

14

We haven't been able to set up a local test environment for all our I/O, particularly modbus, bacnet, mbus etc. Thus we develope everything in prod :smiling_face_with_tear:

15

You point out a lot of pros with js modules, however a subflow reduce visual clutter just as well as a palette package? It's one compressed node that hides all the dirty details and only surface minimum requirements.

16

Well, yes. :wink:

Though, as I've mentioned, if you went the whole hog and created a custom node, you might find other parts of your logic could be incorporated as well.

For production use, my own view is that standardised and relatively stable business logic should be boiled down to actual node.js code rather than being visible in all its gory detail in Node-RED. Leaving the visual part to things that might change or need adjusting. This also, of course, can reduce support issues when customers (inevitably?) mess up your nice visual process.

17

My plan is not just to solve my problem but also to help the community and whoever will search for "subflow sync" in the future. The topic already generated 3+ different approaches to the subject which can be valuable to current and future readers.

I've also found this very interesting by @gregorius, as part of the FlowHub node package:

Personally, I've been diving a bit more into the Node-RED’s Admin API and I am working right now on creating a Subflow which I can export and post here for anybody to use.

It will be configurable with information about remote servers, credentials, and what subflows to be synchronized. It will automatically detect changes and update without a remote server restart, minimal downtime, like an export/import but automated. It will require setting the version of subflows, but I might also automate a version increment. I'll also make a visual tutorial but it'll be as easy as possible to configure.

I will return with that later.

18

Just chipping in here....
Have you thought about packing your subflows as modules?

That is a supported feature, you can then treat them as modules like any other node with proper versioning / dependencies ...

In addition, create a script that then pulls in the latest version (whatever that happens to be) and restarts Node RED?

With a mix of Automated (targeted) Module updates, and then add a pinch of systemctrl to restart Node RED after the update.

Apologies if this thread has moved on, and that idea isn't suitable - I haven't read all comments in detail

19

My introspection package has a GetFlow (retrieve the current flow json file) and SendFlow (send the flow.json to an NR instance) node for interacting with the flow inside NR - might be doing what you're looking to do. Under the hood, both use the NR admin API.

My experience was that I moved, fairly quickly, to using link nodes and entire "flow tabs" for reusability. Subflows were - for me - too rudimentary (don't support context - IIRC) and confusing (using linked nodes inside subflows isn't allowed - IIRC) yet the overhead to sync subflows or flow tabs is basically the same.

Flow tabs being a single tab not the entire NR flow. So functionality is spread using tabs and not subflows (which are also not really first-class NR citizens - IMHO).

But that is also because I had created the flow viewer code which I used to visualise entire flow tabs - so there was an additional motivation for me.

P.S. concerning the NodeDev package you mentioned above, it's best used in combination with this support flow that allows automatic pushes to NPM, GitHub and flows.nodered.org updates as well as npm package creation and downloading.

20

Subflows support context just fine. I prefer to use (sub)flow context cause then it is visible from outside as if the subflow node had node context.