# Multiple node-reds with shared files

**URL:** <https://discourse.nodered.org/t/multiple-node-reds-with-shared-files/85084>\
**Category:** General\
**Created:** [31 January 2024 16:29 UTC](https://discourse.nodered.org/t/multiple-node-reds-with-shared-files/85084 "2024-01-31T16:29:30Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![jmorris644](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/jmorris644/32/31964_2.png) [@jmorris644](https://discourse.nodered.org/u/jmorris644)\
**Post date:** [31 January 2024 16:29 UTC](https://discourse.nodered.org/t/multiple-node-reds-with-shared-files/85084/1 "2024-01-31T16:29:31Z")

</div>

This might be a strange request. I have built my first cluster. The cluster has shared storage. I started two instances of node-red and I had expected that deploying a change via one instance would show up in the other instance with a browser page refresh. This is not the case.

When I restart the instances the changes then show up in both. Is there a config in settings.js or elsewhere where I can synchronize the two instances via the shared flow file?

---

<div class="post-metadata">

**Author:** ![GogoVega](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/gogovega/32/71313_2.png) [@GogoVega](https://discourse.nodered.org/u/GogoVega)\
**Post date:** [31 January 2024 16:46 UTC](https://discourse.nodered.org/t/multiple-node-reds-with-shared-files/85084/2 "2024-01-31T16:46:01Z")

</div>

Because you deploy to one instance, the other instance will only see the changes if you restart it.

If I'm not mistaken you can't do anything in the settings.

You can use a service that restarts the instance as soon as the file is modified (like pm2 with watch) or I recommend FlowFuse which will offer you this query and more

---

<div class="post-metadata">

**Author:** ![jmorris644](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/jmorris644/32/31964_2.png) [@jmorris644](https://discourse.nodered.org/u/jmorris644)\
**Post date:** [31 January 2024 17:03 UTC](https://discourse.nodered.org/t/multiple-node-reds-with-shared-files/85084/3 "2024-01-31T17:03:46Z")

</div>

I will dig further but I did not readily see anything in the FlowFuse docs that led me to believe I can make this work using FlowFuse.

I do like your idea of restarting the instances upon file change. I will look into that.

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [31 January 2024 17:12 UTC](https://discourse.nodered.org/t/multiple-node-reds-with-shared-files/85084/4 "2024-01-31T17:12:33Z")

</div>

> [@jmorris644](#):
>
> I will dig further but I did not readily see anything in the FlowFuse docs that led me to believe I can make this work using FlowFuse

FlowFuse has several ways of achieving this. Here are 2:

1. You can create a node-red instance that is tightly bound to devices. Any time you make a change to the instance you can create a snapshot that gets auto deployed to your "devices" (in your case, your 2 node-reds)
  - [Snapshots • FlowFuse Docs](https://flowfuse.com/docs/user/snapshots/#setting-a-device-target-snapshot)

2. Pipelines: You can create a pipeline for deploying the latest snapshot (or generate a new one) from 1 instance to another - it will auto deploy.
  - [DevOps Pipelines • FlowFuse Docs](https://flowfuse.com/docs/user/devops-pipelines/#devops-pipelines)

How this is achieved by FF is we have created something called a "Device Agent". It runs your Node-REDs and communicates with the FF platform. It can be instructed to reload/restart nodered when new snapshots or settings are applied etc.

---

<div class="post-metadata">

**Author:** ![jmorris644](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/jmorris644/32/31964_2.png) [@jmorris644](https://discourse.nodered.org/u/jmorris644)\
**Post date:** [31 January 2024 17:26 UTC](https://discourse.nodered.org/t/multiple-node-reds-with-shared-files/85084/5 "2024-01-31T17:26:34Z")

</div>

This looks perfect.

Sort of funny, in the last half hour I created a node-red-dev single instance in my cluster alongside my multi-instance node-reds. I then created a job that would shut down the multiple node-red instances, copy the flow file from the node-red-dev, and restart the node-red instances on demand.

I will definitely look at Snapshots and their process though.

---

<div class="post-metadata">

**Author:** ![jmorris644](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/jmorris644/32/31964_2.png) [@jmorris644](https://discourse.nodered.org/u/jmorris644)\
**Post date:** [31 January 2024 18:29 UTC](https://discourse.nodered.org/t/multiple-node-reds-with-shared-files/85084/6 "2024-01-31T18:29:13Z")

</div>

So my little experiment is working with one little issue. After I copy the flow file and restart the node-red instances a Review-Changes process is triggered. Is there a way to mitigate this?

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [31 January 2024 18:58 UTC](https://discourse.nodered.org/t/multiple-node-reds-with-shared-files/85084/7 "2024-01-31T18:58:14Z")

</div>

No. The flows have changed on the server.

Just refresh the browser.

---

<div class="post-metadata">

**Author:** ![GogoVega](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/gogovega/32/71313_2.png) [@GogoVega](https://discourse.nodered.org/u/GogoVega)\
**Post date:** [31 January 2024 18:58 UTC](https://discourse.nodered.org/t/multiple-node-reds-with-shared-files/85084/8 "2024-01-31T18:58:21Z")

</div>

You can ignore it by reload your browser.

Or you can unsubscribe the event (I don't know if it's very appropriate)

```js
RED.comms.unsubscribe("notification/runtime-deploy")

```

(this call must be made each time the editor is opened)

EDIT: will not work because have to give the callback reference

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [31 January 2024 18:59 UTC](https://discourse.nodered.org/t/multiple-node-reds-with-shared-files/85084/9 "2024-01-31T18:59:34Z")

</div>

I believe you won't be able to deploy any changes due to the flows revision differing.

I don't believe this is a wise suggestion.

---

<div class="post-metadata">

**Author:** ![jmorris644](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/jmorris644/32/31964_2.png) [@jmorris644](https://discourse.nodered.org/u/jmorris644)\
**Post date:** [31 January 2024 19:20 UTC](https://discourse.nodered.org/t/multiple-node-reds-with-shared-files/85084/10 "2024-01-31T19:20:32Z")

</div>

Refreshing the browser and even closing the tab and reopening the merge request popoup still comes up.

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [31 January 2024 20:24 UTC](https://discourse.nodered.org/t/multiple-node-reds-with-shared-files/85084/11 "2024-01-31T20:24:01Z")

</div>

Are you sure you are not continually writing to the flows file?

---

<div class="post-metadata">

**Author:** ![jmorris644](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/jmorris644/32/31964_2.png) [@jmorris644](https://discourse.nodered.org/u/jmorris644)\
**Post date:** [31 January 2024 20:55 UTC](https://discourse.nodered.org/t/multiple-node-reds-with-shared-files/85084/12 "2024-01-31T20:55:20Z")

</div>

Yes, it is in a different space. The flow file does not change until I do the file copy at the operating system level.

I will do some more explicit testing, but that as what I am experiencing

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [31 January 2024 21:25 UTC](https://discourse.nodered.org/t/multiple-node-reds-with-shared-files/85084/13 "2024-01-31T21:25:22Z")

</div>

Check the timestamps to make sure.

Are you using the latest version of node red?

---

<div class="post-metadata">

**Author:** ![jmorris644](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/jmorris644/32/31964_2.png) [@jmorris644](https://discourse.nodered.org/u/jmorris644)\
**Post date:** [2 February 2024 14:11 UTC](https://discourse.nodered.org/t/multiple-node-reds-with-shared-files/85084/14 "2024-02-02T14:11:33Z")

</div>

Sorry for the late response. I have been away for a couple of days.

TImestamps are different until copied as expected.

It all seems to work as expected now. I may have had something incorrect in the Nomad definition.

But as a practice, it works very nicely. I have 3 instances of node-red running in a cluster with a single vIP address.

---

<div class="post-metadata">

**Author:** ![meeki007](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/meeki007/32/12499_2.png) [@meeki007](https://discourse.nodered.org/u/meeki007)\
**Post date:** [2 February 2024 15:45 UTC](https://discourse.nodered.org/t/multiple-node-reds-with-shared-files/85084/15 "2024-02-02T15:45:17Z")

</div>

I too have been running a node-red cluster with docker service swarm for a few months and have solved a few issues. The basic knowledge needed in docker, wireguard, shared filesystem (im using glusterfs), and nginx has kept me from even attempting a howto write up.

However you sound like you have the basics down so I will share some of the solutions I came up with.

Issues:  
Storage for any additions. I found issues when using npm packages or user contrib nodes stored to a shared file system.

Binding storage of /data folder for docker but not all the others!

Deploying new flow file to all docker services after I make changes to it.

Solution:  
Every time I need a npm package for all the instances I spin down all my docker instances to 1 and then add the stuff to the docker image. Then I create a new image from this one and then spin it back up.

Cool trick not allot of people ever have to know about docker is I to keep sub folders inside /data from being mounted  
--mount type=volume,destination=/data/.npm   
--mount type=volume,destination=/data/lib   
--mount type=volume,destination=/data/node\_modules   
this keeps them containerized and does not let them be mounted!

Also by using _$ docker service update_ I can refresh all my instances to use the edited flow or if I add new nodes red user contribs / npm file.

the best part of using docker service update is that only one of my instances of node-red in the swarm is down at one time. by using --update-delay 10s \ **when creating my docker service of node-red It will wait 10s between updating each instance to the new flow file**.

Here are my notes: **NOTICE to all** I am unwilling to hold your hand on this. These are my scatter brained notes I keep in a folder so 3 years later I can read them and know where/what I did. They are far from a howto and meant for myself. If you don't understand then you need further remedial education....rather .....start reading. If you have easy or general questions Id be happy to help. If you don't understand this and don't have the want to learn on your own I recommend paying for [flowfuse.com](https://discourse.nodered.org/t/celebrate-10-years-of-node-red/81193)

```auto
####################################
### NODE RED workers ###############
####################################
# THIS IS ALL A PAIN BECAUSE OF REPLICATION AND npm adding nodes!

# create directory to share between workers and manager for this worker
$ mkdir -p $HOME/gluster-mount/docker/cluster/node-red-share/workers

# create directory for instance
$ mkdir -p $HOME/gluster-mount/docker/cluster/node-red-workers/data
note: to remove dir $ sudo rm -r $HOME/gluster-mount/docker/cluster/node-red-workers

# start docker instance
docker service create \
--replicas 1 \
--replicas-max-per-node 1 \
--update-delay 10s \
--update-failure-action pause \
--update-parallelism 1 \
--rollback-delay 10s \
--rollback-failure-action pause \
--rollback-parallelism 1 \
--log-opt max-size=5m \
--log-opt max-file=3 \
--network internalnetwork \
--name "noderedworkers" \
--publish published=xxxx,target=xxxx,protocol=tcp \
--mount type=bind,source="$HOME/gluster-mount/docker/cluster/node-red-workers/data",destination=/data \
nodered/node-red:3.1.3-18

# check that above worked by inspecting logs
$ docker service ls
$ docker service logs -f noderedworkers

# shutdown noderedworkers (from docker manager leider)
$ docker service update --replicas 0 noderedworkers

# edit config file
$ nano $HOME/gluster-mount/docker/cluster/node-red-workers/data/settings.js

#now that we have a running version of node-red with settings we like lets save some files
Create a flows.json file by adding a node and deploying it.

# copy files to backup/server folder:
settings.js

# remove the service
$ docker service rm noderedworkers

# remove the dir
$ sudo rm -r $HOME/gluster-mount/docker/cluster/node-red-workers

# start docker instance ... make sure to set --constraint to the docer leader!!!!

docker service create \
--replicas 1 \
--replicas-max-per-node 1 \
--update-delay 10s \
--update-failure-action pause \
--update-parallelism 1 \
--rollback-delay 10s \
--rollback-failure-action pause \
--rollback-parallelism 1 \
--log-opt max-size=5m \
--log-opt max-file=3 \
--network internalnetwork \
--name "noderedworkers" \
--publish published=xxxx,target=xxxx,protocol=tcp \
--mount type=bind,source="$HOME/gluster-mount/docker/cluster/node-red-workers/data",destination=/data \
--mount type=volume,destination=/data/.npm \
--mount type=volume,destination=/data/lib \
--mount type=volume,destination=/data/node_modules \
--mount type=bind,source="$HOME/gluster-mount/docker/cluster/node-red-share",destination=/data/glusterfs \
--constraint node.hostname==xxxxxxxxxxxxxx.com \
nodered/node-red:3.1.3-18

#check logs and test that its working by loggin in
$ docker service logs -f noderedworkers

# add nodes needed
# nods to add
node-red-contrib-bcrypt
node-red-contrib-crypto-js-dynamic
node-red-contrib-multiple-queue
node-red-contrib-queue-gate
node-red-contrib-string
node-red-contrib-web-worldmap
node-red-node-email

# check that service container is running on leader!!!!
$ docker ps

# Installing npm stuff for use by function node
$ docker exec -it xxxxxxxxxxxxxx /bin/bash
$ cd /data
$ ls
$ npm install --save https://cdn.sheetjs.com/xlsx-0.20.1/xlsx-0.20.1.tgz

# edit settings.js to use module in function node
nano settings.js
---------------------------------------------------

functionGlobalContext: {
         XLSX:require('xlsx'),
    },
---------------------------------------------------

#example usage inside function node

var XLSX = global.get('XLSX');
msg.payload = XLSX.version;
return msg;

# check images
$ docker images docker images --no-trunc

MAKE DANGE SURE NO ONE HAS GET INTO YOUR REGISTY AND CHANGED THESE FILES!!!!!!!
Better yet never pull from image ... edit current runner then push it as a replacemnt then pull from it.

127.0.0.1:xxxx/nodered/customimage 1.0.2 sha256:00fcccccccccccc62cexxxxxxxd46c1f06xxxxxxxxxxxxxxxxxxx42vvvv 3 days ago 634MB

# remember to remove older images ( make sure to keep the last one or two for a rollback!!!)
$ docker image rm
# remove <none> images
$ docker rmi $(docker images -f "dangling=true" -q)

# commit changes (adding nodes) to a new docker image using the container ID
$ docker container commit -a "xxxxxx" -m "Changed default nodered" 940445496ae2 nodered/customimage:1.0.1

#remove current noderedworkers service
$ docker service rm noderedworkers

#create a registy for local images
docker service create \
--replicas 1 \
--replicas-max-per-node 1 \
--update-delay 10s \
--update-failure-action pause \
--update-parallelism 1 \
--rollback-delay 10s \
--rollback-failure-action pause \
--rollback-parallelism 1 \
--log-opt max-size=5m \
--log-opt max-file=3 \
--name "registry" \
--publish xxxx:5000 \
registry:2

#################
# IMPORTANT #
#################
Always disable the registry when not in use spining up/down services.
docker service update --replicas 0 registry

# You can now tag and push your image :
$ docker tag nodered-noderedworkers 127.0.0.1:xxxx/nodered/customimage:1.0.1
$ docker push 127.0.0.1:xxxx/nodered/customimage:1.0.1

NOTE: to create services from that image :
$ docker service create --name myservice localhost:xxxx/myimage:version

# start up a docker service using the new image
docker service create \
--replicas 1 \
--replicas-max-per-node 1 \
--update-delay 10s \
--update-failure-action pause \
--update-parallelism 1 \
--rollback-delay 10s \
--rollback-failure-action pause \
--rollback-parallelism 1 \
--log-opt max-size=5m \
--log-opt max-file=3 \
--network internalnetwork \
--hostname="noderedworkers-{{.Task.Slot}}" \
--name "noderedworkers" \
--publish published=xxxx,target=xxxx,protocol=tcp \
--mount type=bind,source="$HOME/gluster-mount/docker/cluster/node-red-workers/data",destination=/data \
--mount type=volume,destination=/data/.npm \
--mount type=volume,destination=/data/lib \
--mount type=volume,destination=/data/node_modules \
--mount type=bind,source="$HOME/gluster-mount/docker/cluster/node-red-share",destination=/data/glusterfs \
--env NODE_OPTIONS="--max_old_space_size=2048" \
127.0.0.1:xxxx/nodered/customimage:1.0.1

#check logs and test that its working by loggin in
$ docker service logs -f noderedworkers

# start replicas
$ docker service update --replicas 3 noderedworkers

#check logs again to see if all of them spin up and test that its working by loggin in
$ docker service logs -f noderedworkers

# check that services are running
$ docker service ps noderedworkers

### Note when making changes to the flow in nodered make sure to update all dockers services
$ docker service update --force noderedworkers

```

---

<div class="post-metadata">

**Author:** ![jmorris644](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/jmorris644/32/31964_2.png) [@jmorris644](https://discourse.nodered.org/u/jmorris644)\
**Post date:** [3 February 2024 14:51 UTC](https://discourse.nodered.org/t/multiple-node-reds-with-shared-files/85084/16 "2024-02-03T14:51:17Z")

</div>

I was looking real hard at using Swarm. And then I was told by another developer that:

```auto
When Mirabilis bought Docker, they removed all the developers from that work. There is only one actual dev doing any contributions for Swarm any more. It is effectively a dead/abandoned product

```

---

<div class="post-metadata">

**Author:** ![shrickus](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/shrickus/32/517_2.png) [@shrickus](https://discourse.nodered.org/u/shrickus)\
**Post date:** [3 February 2024 15:14 UTC](https://discourse.nodered.org/t/multiple-node-reds-with-shared-files/85084/17 "2024-02-03T15:14:16Z")

</div>

Another possibility is to use the `/admin/flows` apis to "push" new flow.json files to the remote containers from a central node-red server (or even ansible scripting via curl). This would keep the containers from restarting (i'm assuming node-red can restart itself without killing the docker process).

Of course, this requires that each docker instance is read-only, since any changes are saved inside the container. Not sure if that fits with what you are trying to do...

Or what about have a CMD in the dockerfile that pulls the latest version of flows.json from a central git repo before it starts running? Also, I've never tried using the "Projects" feature from a docker container, but it seems like it would have the advantage of getting the latest flows by simply deleting/recreating the containers.

---

<div class="post-metadata">

**Author:** ![meeki007](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/meeki007/32/12499_2.png) [@meeki007](https://discourse.nodered.org/u/meeki007)\
**Post date:** [3 February 2024 18:03 UTC](https://discourse.nodered.org/t/multiple-node-reds-with-shared-files/85084/18 "2024-02-03T18:03:46Z")

</div>

> [@jmorris644](#):
>
> `Mirabilis bought Docker`

> **[Is Docker Swarm Mode EOL?](https://medium.com/@markuman/is-docker-swarm-mode-eol-7a3f316116a3)**
>
> TL;DR

Sure they killed the teams managing the service product. **docker-ce** swarm still is a supported feature Most people follow the trends and many have moved away from docker for its cluster uses.  
If you are going to be managing a large cluster (think banks) kubernetes is the way to go. If your 1 guy trying to do something fun and want to keep it easy Docker Swarm it up!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [3 May 2024 18:04 UTC](https://discourse.nodered.org/t/multiple-node-reds-with-shared-files/85084/19 "2024-05-03T18:04:40Z")

</div>

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.