node-RED Backup Flow

I read a post a few days ago by @Trying_to_learn about losing his node-RED flow, which had taken him no doubt many hours to construct, and as his backup was on the same machine - he lost that too.
So... I've spent a few days looking at off-site storage, and stumbled upon node-red-contrib-dsm which was written by @cflurin complete with an example flow to perform automated backups to dropbox.

dsm

I played about with the configuration, and made some changes to the way that the backups are stored, basically to stop dropbox from filling up, yet conduct a nightly backup and be able to retrieve any one of those backups for 30 days (30 backups). The older ones are automatically weeded by dropbox, so it's 'fit & forget'.
Instead of the backups being named as a timestamp, they are named as the day of the week - Monday_nodered.zip, Tuesday_nodered.zip etc.
Looking at the dropbox web dashboard, you will only see 7 files (the current 7 days), but if you go to 'Version History', you can see & download the previous versions of that day, so for example every Monday's backup for the past 30 days.

Anyway, here's the flow, it uses node-red-contrib-dsm, node-red-node-dropbox and a couple of inject nodes.
Also you need to ensure that zip is installed, try zip --help and if help isn't displayed, then install it as sudo apt-get install zip unzip.
The flow should work out of the box, once you've configured the dropbox node (leave the Filename & Local Filename boxes empty).
PS The inject node is set to run the flow 3.15am every night.

NOTE: See edit 17/4/21 below for +v1.3.0 version

[{"id":"7868f6a6.feb728","type":"dropbox out","z":"4487e413.bb781c","dropbox":"","filename":"","localFilename":"","name":"","x":430,"y":780,"wires":[]},{"id":"72f00992.3036f8","type":"inject","z":"4487e413.bb781c","name":"backup","props":[{"p":"payload","v":"","vt":"date"},{"p":"topic","v":"start","vt":"string"}],"repeat":"","crontab":"15 03 * * *","once":false,"onceDelay":0.1,"topic":"start","payload":"","payloadType":"date","x":132,"y":760,"wires":[["b0bc8692.a8e948"]]},{"id":"8b602104.6656a","type":"inject","z":"4487e413.bb781c","name":"reset","repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"reset","payload":"","payloadType":"date","x":140,"y":800,"wires":[["b0bc8692.a8e948"]]},{"id":"b0bc8692.a8e948","type":"dsm","z":"4487e413.bb781c","name":"backup","sm_config":"{\n    \"currentState\": \"step1\",\n    \"states\": {\n        \"step1\": {\n            \"start\": \"step2\",\n            \"reset\": \"step1\"\n        },\n        \"step2\": {\n            \"zip\": \"step3\",\n            \"reset\": \"step1\"\n        },\n        \"step3\": {\n            \"upload\": \"step1\",\n            \"reset\": \"step1\"\n        }\n    },\n    \"data\": {\n    },\n    \"methods\": {\n        \"init\": [\n            \"sm.udir = RED.settings.userDir + '/';\",\n            \"sm.exec = require('child_process').exec;\",\n            \"sm.hostname = require('os').hostname();\",\n            \"sm.unlink = require('fs').unlink;\"\n        ],\n        \"start\": [\n            \"/* delete old zip file */\",\n            \"var zipfile = sm.udir+'node-red.zip';\",\n            \n            \"sm.fill = 'grey';\",\n            \"sm.text = 'deleting';\",\n            \"output = false;\",\n\n            \"sm.unlink(zipfile, function (err) {\",\n            \"   if (err) {\",\n            \"       node.warn('no file '+zipfile);\",\n            \"   }\",\n            \"   resume('zip', msg);\",\n            \"});\"\n        ],\n        \"zip\": [\n            \"var pre = ' ' + sm.udir;\",\n            \"var cmd = 'zip -r';\",\n            \"cmd += pre + 'node-red.zip';\",\n            \"cmd += pre + 'flows*.json';\",\n            \"cmd += pre + '.config*.json';\",\n            \"cmd += pre + 'settings.js';\",\n            \"cmd += pre + 'package.json';\",\n            \"cmd += pre + 'package-lock.json';\",\n            \"cmd += pre + 'lib/*';\",\n\n            \"sm.fill = 'grey';\",\n            \"sm.text = 'zipping';\",\n            \"output = false;\",\n            \n            \"sm.exec(cmd, function(error, stdout, stderr) {\",\n            \"   if (error) {\",\n            \"       node.warn(error);\",\n            \"   } else {\",\n            \"       resume('upload', msg);\",\n            \"   }\",\n            \"});\"\n        ],\n        \"upload\": [\n            \"sm.ts = new Date();\",\n            \"sm.days = ['Sunday','Monday','Tuesday','Wednesday','Thursday','Friday','Saturday'];\",\n            \"sm.day = sm.days[sm.ts.getDay()];\",\n            \"msg.filename = sm.day + '_nodered.zip';\",\n            \"msg.localFilename = sm.udir+'node-red.zip';\",\n            \"sm.fill = 'green';\",\n            \"sm.text = sm.day;\"\n        ],\n        \"reset\": [\n            \"sm.fill = 'grey';\",\n            \"sm.text = 'reset';\",\n            \"output = false;\"\n        ],\n        \"status\": {\n            \"fill\": {\n                \"get\": \"sm.fill\"\n            },\n            \"shape\": \"dot\",\n            \"text\": {\n                \"get\": \"sm.text\"\n            }\n        }\n    }\n}","x":285,"y":780,"wires":[["7868f6a6.feb728"]]}]

EDIT 20/10/2020 - Flow amended to ensure that all of the config files are backed up following the changes to node-RED in v1.2.0

EDIT 17/4/2021 - Flow updated below to include the additional package.json contained within the externalModules folder, which was introduced in v1.3.0

[{"id":"7868f6a6.feb728","type":"dropbox out","z":"b3b413d1.05b1b","dropbox":"","filename":"","localFilename":"","name":"","x":470,"y":1605,"wires":[]},{"id":"72f00992.3036f8","type":"inject","z":"b3b413d1.05b1b","name":"backup","props":[{"p":"payload","v":"","vt":"date"},{"p":"topic","v":"start","vt":"string"}],"repeat":"","crontab":"15 03 * * *","once":false,"onceDelay":0.1,"topic":"start","payload":"","payloadType":"date","x":172,"y":1585,"wires":[["b0bc8692.a8e948"]]},{"id":"8b602104.6656a","type":"inject","z":"b3b413d1.05b1b","name":"reset","repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"reset","payload":"","payloadType":"date","x":180,"y":1625,"wires":[["b0bc8692.a8e948"]]},{"id":"b0bc8692.a8e948","type":"dsm","z":"b3b413d1.05b1b","name":"backup","sm_config":"{\n    \"currentState\": \"step1\",\n    \"states\": {\n        \"step1\": {\n            \"start\": \"step2\",\n            \"reset\": \"step1\"\n        },\n        \"step2\": {\n            \"zip\": \"step3\",\n            \"reset\": \"step1\"\n        },\n        \"step3\": {\n            \"upload\": \"step1\",\n            \"reset\": \"step1\"\n        }\n    },\n    \"data\": {\n    },\n    \"methods\": {\n        \"init\": [\n            \"sm.udir = RED.settings.userDir + '/';\",\n            \"sm.exec = require('child_process').exec;\",\n            \"sm.hostname = require('os').hostname();\",\n            \"sm.unlink = require('fs').unlink;\"\n        ],\n        \"start\": [\n            \"/* delete old zip file */\",\n            \"var zipfile = sm.udir+'node-red.zip';\",\n            \n            \"sm.fill = 'grey';\",\n            \"sm.text = 'deleting';\",\n            \"output = false;\",\n\n            \"sm.unlink(zipfile, function (err) {\",\n            \"   if (err) {\",\n            \"       node.warn('no file '+zipfile);\",\n            \"   }\",\n            \"   resume('zip', msg);\",\n            \"});\"\n        ],\n        \"zip\": [\n            \"var pre = ' ' + sm.udir;\",\n            \"var cmd = 'zip -r';\",\n            \"cmd += pre + 'node-red.zip';\",\n            \"cmd += pre + 'flows*.json';\",\n            \"cmd += pre + '.config*.json';\",\n            \"cmd += pre + 'settings.js';\",\n            \"cmd += pre + 'package.json';\",\n            \"cmd += pre + 'package-lock.json';\",\n            \"cmd += pre + 'externalModules/package.json';\",\n            \"cmd += pre + 'externalModules/package-lock.json';\",\n            \"cmd += pre + 'lib/*';\",\n\n            \"sm.fill = 'grey';\",\n            \"sm.text = 'zipping';\",\n            \"output = false;\",\n            \n            \"sm.exec(cmd, function(error, stdout, stderr) {\",\n            \"   if (error) {\",\n            \"       node.warn(error);\",\n            \"   } else {\",\n            \"       resume('upload', msg);\",\n            \"   }\",\n            \"});\"\n        ],\n        \"upload\": [\n            \"sm.ts = new Date();\",\n            \"sm.days = ['Sunday','Monday','Tuesday','Wednesday','Thursday','Friday','Saturday'];\",\n            \"sm.day = sm.days[sm.ts.getDay()];\",\n            \"msg.filename = sm.day + '_nodered.zip';\",\n            \"msg.localFilename = sm.udir+'node-red.zip';\",\n            \"sm.fill = 'green';\",\n            \"sm.text = sm.day;\"\n        ],\n        \"reset\": [\n            \"sm.fill = 'grey';\",\n            \"sm.text = 'reset';\",\n            \"output = false;\"\n        ],\n        \"status\": {\n            \"fill\": {\n                \"get\": \"sm.fill\"\n            },\n            \"shape\": \"dot\",\n            \"text\": {\n                \"get\": \"sm.text\"\n            }\n        }\n    }\n}","x":325,"y":1605,"wires":[["7868f6a6.feb728"]]}]

EDIT 21/7/2021 - Flow updated below to accommodate changes made in node-RED v2.0.
The externalModules is now depreciated, and the file .config.modules.json has been added to the backup set.

[{"id":"7868f6a6.feb728","type":"dropbox out","z":"b3b413d1.05b1b","dropbox":"","filename":"","localFilename":"","name":"","x":440,"y":1380,"wires":[]},{"id":"72f00992.3036f8","type":"inject","z":"b3b413d1.05b1b","name":"backup","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"15 03 * * *","once":false,"onceDelay":0.1,"topic":"start","payloadType":"date","x":137,"y":1360,"wires":[["b0bc8692.a8e948"]]},{"id":"8b602104.6656a","type":"inject","z":"b3b413d1.05b1b","name":"reset","repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"reset","payload":"","payloadType":"date","x":145,"y":1400,"wires":[["b0bc8692.a8e948"]]},{"id":"b0bc8692.a8e948","type":"dsm","z":"b3b413d1.05b1b","name":"backup","sm_config":"{\n    \"currentState\": \"step1\",\n    \"states\": {\n        \"step1\": {\n            \"start\": \"step2\",\n            \"reset\": \"step1\"\n        },\n        \"step2\": {\n            \"zip\": \"step3\",\n            \"reset\": \"step1\"\n        },\n        \"step3\": {\n            \"upload\": \"step1\",\n            \"reset\": \"step1\"\n        }\n    },\n    \"data\": {\n    },\n    \"methods\": {\n        \"init\": [\n            \"sm.udir = RED.settings.userDir + '/';\",\n            \"sm.exec = require('child_process').exec;\",\n            \"sm.hostname = require('os').hostname();\",\n            \"sm.unlink = require('fs').unlink;\"\n        ],\n        \"start\": [\n            \"/* delete old zip file */\",\n            \"var zipfile = sm.udir+'node-red.zip';\",\n            \n            \"sm.fill = 'grey';\",\n            \"sm.text = 'deleting';\",\n            \"output = false;\",\n\n            \"sm.unlink(zipfile, function (err) {\",\n            \"   if (err) {\",\n            \"       node.warn('no file '+zipfile);\",\n            \"   }\",\n            \"   resume('zip', msg);\",\n            \"});\"\n        ],\n        \"zip\": [\n            \"var pre = ' ' + sm.udir;\",\n            \"var cmd = 'zip -r';\",\n            \"cmd += pre + 'node-red.zip';\",\n            \"cmd += pre + 'flows*.json';\",\n            \"cmd += pre + '.config*.json';\",\n            \"cmd += pre + 'settings.js';\",\n            \"cmd += pre + 'package.json';\",\n            \"cmd += pre + 'package-lock.json';\",\n            \"cmd += pre + '.config.modules.json';\",\n            \"cmd += pre + 'lib/*';\",\n\n            \"sm.fill = 'grey';\",\n            \"sm.text = 'zipping';\",\n            \"output = false;\",\n            \n            \"sm.exec(cmd, function(error, stdout, stderr) {\",\n            \"   if (error) {\",\n            \"       node.warn(error);\",\n            \"   } else {\",\n            \"       resume('upload', msg);\",\n            \"   }\",\n            \"});\"\n        ],\n        \"upload\": [\n            \"sm.ts = new Date();\",\n            \"sm.days = ['Sunday','Monday','Tuesday','Wednesday','Thursday','Friday','Saturday'];\",\n            \"sm.day = sm.days[sm.ts.getDay()];\",\n            \"msg.filename = sm.day + '_nodered.zip';\",\n            \"msg.localFilename = sm.udir+'node-red.zip';\",\n            \"sm.fill = 'green';\",\n            \"sm.text = sm.day;\"\n        ],\n        \"reset\": [\n            \"sm.fill = 'grey';\",\n            \"sm.text = 'reset';\",\n            \"output = false;\"\n        ],\n        \"status\": {\n            \"fill\": {\n                \"get\": \"sm.fill\"\n            },\n            \"shape\": \"dot\",\n            \"text\": {\n                \"get\": \"sm.text\"\n            }\n        }\n    }\n}","x":290,"y":1380,"wires":[["7868f6a6.feb728"]]}]
14 Likes

@Paul-Reed Nice, but I have a suggestion, why not run the flow every hour or half hour and compare it to the previous version. If they are different, save the new one then while developing you would have multiple backups to fall back to if you mess something up (not like I've ever done that...more than a few dozen or more times :rofl:)

As the DSM saves the zip file in the /node-red user directory, you would probably need to regularly check the size of the zip file, and if it's changed, then upload that file to dropbox.
However, I like this because of its simplicity, and maybe if it starts to get too complicated, some users may be deterred from using it.

Users could always trigger a backup at anytime, just by clicking the inject node.

2 Likes

I'm using the version from the dsm-wiki for months on 3 Rpis. Two of them are running without any changes for weeks, so I only manually backup node-red by changes.

1 Like

This topic was automatically closed after 60 days. New replies are no longer allowed.

Just came here following the backup dicussion in another thread and decided to try it out on my win10 machine

It didn't work because win10 has no command line tool called zip

But it does have one called tar so I just did a simple mode of the dsm node and it works fine :slight_smile:
image

here is the win10 alternative dsm node only

[{"id":"20a4ec54.6c6454","type":"dsm","z":"fa1b22c7.a8823","name":"backup","sm_config":"{\n    \"currentState\": \"step1\",\n    \"states\": {\n        \"step1\": {\n            \"start\": \"step2\",\n            \"reset\": \"step1\"\n        },\n        \"step2\": {\n            \"zip\": \"step3\",\n            \"reset\": \"step1\"\n        },\n        \"step3\": {\n            \"upload\": \"step1\",\n            \"reset\": \"step1\"\n        }\n    },\n    \"data\": {\n    },\n    \"methods\": {\n        \"init\": [\n            \"sm.udir = RED.settings.userDir + '/';\",\n            \"sm.exec = require('child_process').exec;\",\n            \"sm.hostname = require('os').hostname();\",\n            \"sm.unlink = require('fs').unlink;\"\n        ],\n        \"start\": [\n            \"/* delete old zip file */\",\n            \"var zipfile = sm.udir+'node-red.zip';\",\n            \n            \"sm.fill = 'grey';\",\n            \"sm.text = 'deleting';\",\n            \"output = false;\",\n\n            \"sm.unlink(zipfile, function (err) {\",\n            \"   if (err) {\",\n            \"       node.warn('no file '+zipfile);\",\n            \"   }\",\n            \"   resume('zip', msg);\",\n            \"});\"\n        ],\n        \"zip\": [\n            \"var pre = ' ' + sm.udir;\",\n            \"var cmd = 'tar.exe -a -c -f';\",\n            \"cmd += pre + 'node-red.zip';\",\n            \"cmd += pre + 'flows*.json';\",\n            \"cmd += pre + '.config.json';\",\n            \"cmd += pre + '.sessions.json';\",\n            \"cmd += pre + 'settings.js';\",\n            \"cmd += pre + 'package.json';\",\n            \"cmd += pre + 'package-lock.json';\",\n            \"cmd += pre + 'lib/*';\",\n\n            \"sm.fill = 'grey';\",\n            \"sm.text = 'zipping';\",\n            \"output = false;\",\n            \n            \"sm.exec(cmd, function(error, stdout, stderr) {\",\n            \"   if (error) {\",\n            \"       node.warn(error);\",\n            \"   } else {\",\n            \"       resume('upload', msg);\",\n            \"   }\",\n            \"});\"\n        ],\n        \"upload\": [\n            \"sm.ts = new Date();\",\n            \"sm.days = ['Sunday','Monday','Tuesday','Wednesday','Thursday','Friday','Saturday'];\",\n            \"sm.day = sm.days[sm.ts.getDay()];\",\n            \"msg.filename = sm.day + '_nodered.zip';\",\n            \"msg.localFilename = sm.udir+'node-red.zip';\",\n            \"sm.fill = 'green';\",\n            \"sm.text = sm.day;\"\n        ],\n        \"reset\": [\n            \"sm.fill = 'grey';\",\n            \"sm.text = 'reset';\",\n            \"output = false;\"\n        ],\n        \"status\": {\n            \"fill\": {\n                \"get\": \"sm.fill\"\n            },\n            \"shape\": \"dot\",\n            \"text\": {\n                \"get\": \"sm.text\"\n            }\n        }\n    }\n}","x":355,"y":160,"wires":[["44702648.1f2418"]]}]
2 Likes

Just a note, the user may need to install zip. All my installs have unzip but none had zip

Hi @Buckskin, I've moved your post here because that issue was raised earlier today by @cymplecy, and also other users trying this solution will read it.

Cheers, thanks for the flow. Works well.

I have no clue about the DSM node so I have changed the save filename (to distinguish WHICH Pi I am backing up) in a Change node between it and the Dropbox node. Is there any way of specifying something in the Inject node to do the same thing?

Probably something that the DSM node author @cflurin may be able to answer.
I'm not sure that the msg.payload is accessible from within the node itself.

I've just tried adding "sm.id = msg.payload;", but it didn't seem to like it!

The flow did its first auto backup early this morning :slight_smile:
image

I then managed to modify it to prefix the file name with the msg.payload from the inject node
image

image

Modified DSM node only

[{"id":"20a4ec54.6c6454","type":"dsm","z":"fa1b22c7.a8823","name":"backup","sm_config":"{\n    \"currentState\": \"step1\",\n    \"states\": {\n        \"step1\": {\n            \"start\": \"step2\",\n            \"reset\": \"step1\"\n        },\n        \"step2\": {\n            \"zip\": \"step3\",\n            \"reset\": \"step1\"\n        },\n        \"step3\": {\n            \"upload\": \"step1\",\n            \"reset\": \"step1\"\n        }\n    },\n    \"data\": {\n    },\n    \"methods\": {\n        \"init\": [\n            \"sm.udir = RED.settings.userDir + '/';\",\n            \"sm.exec = require('child_process').exec;\",\n            \"sm.hostname = require('os').hostname();\",\n            \"sm.unlink = require('fs').unlink;\"\n        ],\n        \"start\": [\n            \"/* delete old zip file */\",\n            \"var zipfile = sm.udir+'node-red.zip';\",\n            \n            \"sm.fill = 'grey';\",\n            \"sm.text = 'deleting';\",\n            \"output = false;\",\n\n            \"sm.unlink(zipfile, function (err) {\",\n            \"   if (err) {\",\n            \"       node.warn('no file '+zipfile);\",\n            \"   }\",\n            \"   resume('zip', msg);\",\n            \"});\"\n        ],\n        \"zip\": [\n            \"var pre = ' ' + sm.udir;\",\n            \"var cmd = 'tar -a -c -f';\",\n            \"cmd += pre + 'node-red.zip';\",\n            \"cmd += pre + 'flows*.json';\",\n            \"cmd += pre + '.config.json';\",\n            \"cmd += pre + '.sessions.json';\",\n            \"cmd += pre + 'settings.js';\",\n            \"cmd += pre + 'package.json';\",\n            \"cmd += pre + 'package-lock.json';\",\n            \"cmd += pre + 'lib/*';\",\n\n            \"sm.fill = 'grey';\",\n            \"sm.text = 'zipping';\",\n            \"output = false;\",\n            \n            \"sm.exec(cmd, function(error, stdout, stderr) {\",\n            \"   if (error) {\",\n            \"       node.warn(error);\",\n            \"   } else {\",\n            \"       resume('upload', msg);\",\n            \"   }\",\n            \"});\"\n        ],\n        \"upload\": [\n            \"sm.ts = new Date();\",\n            \"sm.days = ['Sunday','Monday','Tuesday','Wednesday','Thursday','Friday','Saturday'];\",\n            \"sm.day = sm.days[sm.ts.getDay()];\",\n            \"msg.filename = msg.payload + '_' + sm.day + '_nodered.zip';\",\n            \"msg.localFilename = sm.udir + 'node-red.zip';\",\n            \"sm.fill = 'green';\",\n            \"sm.text = sm.day;\"\n        ],\n        \"reset\": [\n            \"sm.fill = 'grey';\",\n            \"sm.text = 'reset';\",\n            \"output = false;\"\n        ],\n        \"status\": {\n            \"fill\": {\n                \"get\": \"sm.fill\"\n            },\n            \"shape\": \"dot\",\n            \"text\": {\n                \"get\": \"sm.text\"\n            }\n        }\n    }\n}","x":355,"y":160,"wires":[["44702648.1f2418"]]}]
2 Likes

I think that I must have been overthinking it last night :shushing_face:

1 Like

You probably added it in the init method.
This method is called once on deploy/start, as the input isn't triggered the msg object is undefined. All other methods can access the msg object (payload, topic etc.).

1 Like

JFI I tried my modified DSM node out on a Pi but it kept erroring out

The tar command didn't like the fact that there was no .sessions.json file on my Pi

It worked fine once I deleted it from the list of backup files

Is it important to backup .sessions.json? what is that file for?

I did give some thought to removing that, because as far as I'm aware, it's a file that keeps users logged in, with the assumption that if deleted, all users would again need to login, and the file would be recreated (I'm happy to be corrected @dceejay ).
If so, maybe restoring the .sessions.json could create more problems than not?

But... it was in @cflurin's original DSM configuration, and this post, which persuaded me to keep it in.
Maybe it should go??

As far as I know the .session.json can be omitted.
I thought it would not be wrong to backup .session.json.
At restore you can decide to use it or not.

What is the consensus re .sessions.json, does it stay, or shall I remove it from the example flow config that I posted in post #1?

ie. Is it necessary for .sessions.json to form part of a node-RED backup archive.

No responses... so it has now been removed from the example flow above.

Missed your original question. The sessions file contains the active login tokens for the editor. No need to back it up.

2 Likes