UPDATED: 2021-02-28 - Fixed the script so that it works properly - reminder to self, don't use relative paths when working with CRON!
Hi again all, on a roll today.
This time I finally got round to writing a backup script so that I don't have to do it by hand. I've usually done this with a remote tool in the past but I wanted some extra cover and something I can run manually as well if I need an extra backup.
The script here is written as a daily rotating backup so there are some things that you might want to do, I've listed some at the end.
You will need to have rsync installed but nothing else is needed other than CRON to run on a daily schedule (you could do that from within Node-RED if you wanted to).
However, this version also uses mosquitto_pub to send notifications on success or failure. Remove if you don't want that. You could instead use CURL to trigger a Node-RED endpoint.
#! /usr/bin/env bash
# Redirect stdout to syslog - show with: `sudo journalctl -t nrmain-backup`
exec 1> >(logger -t nrmain-backup -p local0.info)
# redirect stderr to syslog
exec 2> >(logger -t nrmain-backup -p local0.err)
# --- SET THESE TO THE CORRECT LOCATIONS --- #
SOURCE_PATH=/home/home/nrmain
DEST_PATH=/home/home/nrmain-backup
# ------------------------------------------ #
STARTDATE=$(date +'%Y-%m-%d %T')
echo " "
echo "Starting daily backup of $SOURCE_PATH/ to $DEST_PATH/ ..."
echo "Rotating snapshots ..."
# Delete oldest daily backup
if [ -d $DEST_PATH/daily.7 ] ; then
echo " Deleting oldest daily backup $DEST_PATH/daily.7"
rm -rf $DEST_PATH/daily.7
fi
# Shift all other daily backups ahead one day
for OLD in 6 5 4 3 2 1 ; do
if [ -d $DEST_PATH/daily.$OLD ] ; then
NEW=$(($OLD+1))
echo " Moving $DEST_PATH/daily.$OLD to $DEST_PATH/daily.$NEW"
# Backup last date
# ISSUE: touch does not support options on synology (busybox) system
touch $DEST_PATH/.timestamp -r $DEST_PATH/daily.$OLD
mv $DEST_PATH/daily.$OLD $DEST_PATH/daily.$NEW
# Restore timestamp
touch $DEST_PATH/daily.$NEW -r $DEST_PATH/.timestamp
fi
done
# Copy hardlinked snapshot of level 0 to level 1 (before updating 0 via rsync)
if [ -d $DEST_PATH/daily.0 ] ; then
echo " Copying hardlinks from $DEST_PATH/daily.0 to $DEST_PATH/daily.1"
cp -al $DEST_PATH/daily.0 $DEST_PATH/daily.1
fi
echo "Finished rotating snapshots ..."
if ! [ -d $DEST_PATH/daily.0 ] ; then
mkdir -p $DEST_PATH/daily.0
fi
# Set today's date on the current backup folder
touch $DEST_PATH/daily.0
ENDDATE=$(date --iso-8601=s)
# Back up
echo "Performing rsync backup ..."
rsync --archive --hard-links --delete --delete-excluded \
--exclude 'node_modules' --exclude 'data/node_modules' \
$SOURCE_PATH/ $DEST_PATH/daily.0
# Validate return code
# 0 = no error,
# 24 is fine, happens when files are being touched during sync (logs etc)
# all other codes are fatal -- see man (1) rsync
if ! [ $? = 24 -o $? = 0 ] ; then
echo "Fatal: Node-RED daily backup finished with errors!"
mosquitto_pub -r -t services/nrmainbackup/daily/fail -m $ENDDATE
else
echo "Finished Node-RED daily backup, no errors."
mosquitto_pub -r -t services/nrmainbackup/daily/success -m $ENDDATE
fi
# Sync disks to make sure data is written to disk
sync
#EOF
Some things you might want to do differently:
This version is set up to work with my alternate installer and assumes that you are running Node-RED with a user account. The folder containing node-red is ~/nrmain, the userDir folder is ~/nrmain/data. You might want to adjust paths.
Have more than 7d of backups. Maybe a second script that rotates monthly.
And as a bonus, here is the monthly script that I'm using:
#! /usr/bin/env bash
# Monthly backup for node-red
# Redirect stdout to syslog - show with: `sudo journalctl -t nrmain-backup`
exec 1> >(logger -t nrmain-backup -p local0.info)
# redirect stderr to syslog
exec 2> >(logger -t nrmain-backup -p local0.err)
# --- SET THESE TO THE CORRECT LOCATIONS --- #
SOURCE_PATH=/home/home/nrmain
DEST_PATH=/home/home/nrmain-backup
# ------------------------------------------ #
STARTDATE=$(date +'%Y-%m-%d %T')
echo " "
echo "Starting monthly backup of $SOURCE_PATH/ to $DEST_PATH/ ..."
echo "Rotating snapshots ..."
# Delete oldest daily backup
if [ -d $DEST_PATH/monthly.12 ] ; then
echo "Deleting oldest monthly backup $DEST_PATH/monthly.12"
rm -rf $DEST_PATH/monthly.12
fi
# Shift all other daily backups ahead one day
for OLD in 11 10 9 8 7 6 5 4 3 2 1 ; do
if [ -d $DEST_PATH/monthly.$OLD ] ; then
NEW=$(($OLD+1))
echo "Moving $DEST_PATH/monthly.$OLD to $DEST_PATH/monthly.$NEW"
# Backup last date
# ISSUE: touch does not support options on synology (busybox) system
touch $DEST_PATH/.timestamp -r $DEST_PATH/monthly.$OLD
mv $DEST_PATH/monthly.$OLD $DEST_PATH/monthly.$NEW
# Restore timestamp
touch $DEST_PATH/monthly.$NEW -r $DEST_PATH/.timestamp
fi
done
# Copy hardlinked snapshot of level 0 to level 1 (before updating 0 via rsync)
if [ -d $DEST_PATH/monthly.0 ] ; then
echo "Copying hardlinks from $DEST_PATH/monthly.0 to $DEST_PATH/monthly.1"
cp -al $DEST_PATH/monthly.0 $DEST_PATH/monthly.1
fi
echo "Finished rotating snapshots ..."
if ! [ -d $DEST_PATH/monthly.0 ] ; then
mkdir -p $DEST_PATH/monthly.0
fi
# Set today's date on the current backup folder
touch $DEST_PATH/monthly.0
ENDDATE=$(date --iso-8601=s)
# Back up
echo "Performing rsync backup ..."
rsync --archive --hard-links --delete --delete-excluded \
--exclude 'node_modules' --exclude 'data/node_modules' \
$SOURCE_PATH/ $DEST_PATH/monthly.0
# Validate return code
# 0 = no error,
# 24 is fine, happens when files are being touched during sync (logs etc)
# all other codes are fatal -- see man (1) rsync
if ! [ $? = 24 -o $? = 0 ] ; then
echo "Fatal: Node-RED monthly backup finished with errors!"
#curl --insecure -I 'https://localhost:1880/nrnotify'
mosquitto_pub -r -t services/nrmainbackup/monthly/fail -m $ENDDATE
else
echo "Finished Node-RED monthly backup, no errors."
#curl --insecure -I 'https://localhost:1880/nrnotify'
mosquitto_pub -r -t services/nrmainbackup/monthly/success -m $ENDDATE
fi
# Sync disks to make sure data is written to disk
sync
#EOF
@TotallyInformation - a few years now since you posted this... but before I install them...
Are these still your current backup scripts, and are they still working OK for you?
Nope, had no reason to change them at all! They do what I need them to do. I think I might have recovered something once in all that time - Node-RED is just too darned stable!
Just to double-check though, here is the current daily script:
#! /usr/bin/env bash
# Redirect stdout to syslog -
# show with: `sudo journalctl -t nrmain-backup`
# or `sudo cat /var/log/syslog | grep nrmain-backup`
exec 1> >(logger -t nrmain-backup -p local0.info)
# redirect stderr to syslog
exec 2> >(logger -t nrmain-backup -p local0.err)
# --- SET THESE TO THE CORRECT LOCATIONS --- #
NR_SERVICE_NAME=nrmain
NR_SOURCE_PATH=/home/home/nrmain
NR_DEST_PATH=/home/home/nrmain-backup
# ------------------------------------------ #
STARTDATE=$(date +'%Y-%m-%d %T')
echo " "
echo "Starting daily backup of $NR_SOURCE_PATH/ to $NR_DEST_PATH/ ..."
echo "Rotating snapshots ..."
# Delete oldest daily backup
if [ -d $NR_DEST_PATH/daily.7 ] ; then
echo " Deleting oldest daily backup $NR_DEST_PATH/daily.7"
# The slow but understandable way:
#rm -rf $NR_DEST_PATH/daily.7
# The faster way (needs an empty folder)
rsync -rd --delete $NR_DEST_PATH/empty/ $NR_DEST_PATH/daily.7/
fi
# Shift all other daily backups ahead one day
for OLD in 6 5 4 3 2 1 ; do
if [ -d $NR_DEST_PATH/daily.$OLD ] ; then
NEW=$(($OLD+1))
echo " Moving $NR_DEST_PATH/daily.$OLD to $NR_DEST_PATH/daily.$NEW"
# Backup last date
# ISSUE: touch does not support options on synology (busybox) system
touch $NR_DEST_PATH/.dtimestamp -r $NR_DEST_PATH/daily.$OLD
mv $NR_DEST_PATH/daily.$OLD $NR_DEST_PATH/daily.$NEW
# Restore timestamp
touch $NR_DEST_PATH/daily.$NEW -r $NR_DEST_PATH/.dtimestamp
fi
done
# Copy hardlinked snapshot of level 0 to level 1 (before updating 0 via rsync)
if [ -d $NR_DEST_PATH/daily.0 ] ; then
echo " Copying hardlinks from $NR_DEST_PATH/daily.0 to $NR_DEST_PATH/daily.1"
cp -al $NR_DEST_PATH/daily.0 $NR_DEST_PATH/daily.1
fi
echo "Finished rotating snapshots ..."
if ! [ -d $NR_DEST_PATH/daily.0 ] ; then
mkdir -p $NR_DEST_PATH/daily.0
fi
# Set today's date on the current backup folder
touch $NR_DEST_PATH/daily.0
ENDDATE=$(date --iso-8601=s)
# Back up
echo "Performing rsync backup ..."
rsync --archive --hard-links --delete --delete-excluded \
--exclude 'node_modules' --exclude 'data/node_modules' --exclude 'data/externalModules/node_modules' \
$NR_SOURCE_PATH/ $NR_DEST_PATH/daily.0
# Validate return code
# 0 = no error,
# 24 is fine, happens when files are being touched during sync (logs etc)
# all other codes are fatal -- see man (1) rsync
# You can output the result to MQTT or to a Node-RED http-in endpoint
if ! [ $? = 24 -o $? = 0 ] ; then
echo "Fatal: Node-RED daily backup finished with errors!"
#curl --insecure -I 'https://localhost:1880/nrnotify?type=backup&schedule=daily&result=fail'
mosquitto_pub -r -t services/$NR_SERVICE_NAME/backup/daily/fail -m $ENDDATE
else
echo "Finished Node-RED daily backup, no errors."
#curl --insecure -I 'https://localhost:1880/nrnotify?type=backup&schedule=daily&result=success'
mosquitto_pub -r -t services/$NR_SERVICE_NAME/backup/daily/success -m $ENDDATE
fi
# Sync disks to make sure data is written to disk
sync
#EOF
If that is different, let me know and I'll re-share the weekly and monthly scripts.
I should note that I also use the Backup tool on my NAS to make an encrypted backup of the backups and that backup is, itself, further copied to a spare OneDrive. So I have the 2 local copies and a cloud copy.
I'm aware that something called logrotate exists but I've never actively used it.
I'm also not a BASH expert nor particularly an RSYNC one either. The backup scripts were based on other examples on the Internet.
Honestly, they are probably massive overkill for most people. But they work and don't seem to take a lot of resources so they do the job and that's all I need. A week of daily backups, a month of weekly backups and a year of monthly backups.
If I were to do it over, I'd probably simply do a daily backup to NAS and let the NAS's version history take care of the rest. But as I say, it works so there's no incentive to change it.
Another issue I've found is that because the backup directories are renamed daily.0 > daily.1 etc, they all have the same timestamp, so you can't determine when the backup was created..
I see in the script, an attempt was made to restore the directory timestamps, to their original time using touch, but that doesn't seem to be working.
I've made a few attempts to fix it, but I'm not that familiar with Bash, although I think that .dtimestamp never gets the timestamp from daily.$OLD as it never changes. (Line 1 of above code).
Possible because touch -r only works with files & not directories?
Ah yes, the script in the first post has; touch $DEST_PATH/monthly.$NEW -r $DEST_PATH/.timestamp
..but @TotallyInformation's later revision has; touch $NR_DEST_PATH/daily.$NEW -r $NR_DEST_PATH/.dtimestamp
But even editing to touch $DEST_PATH/monthly.$NEW -r $DEST_PATH/.timestamp and running some minutes apart, I get;
-rwxr-xr-x 1 pi pi 3139 Jan 4 11:58 backupScript.sh
drwxr-xr-x 5 pi pi 4096 Jan 4 03:15 daily.0
drwxr-xr-x 5 pi pi 4096 Jan 4 03:15 daily.1
drwxr-xr-x 5 pi pi 4096 Jan 4 03:15 daily.2
drwxr-xr-x 5 pi pi 4096 Jan 4 03:15 daily.3
drwxr-xr-x 5 pi pi 4096 Jan 4 03:15 daily.4
-rw-r--r-- 1 pi pi 0 Jan 4 03:15 .dtimestamp
drwxr-xr-x 2 pi pi 4096 Jan 4 11:55 empty
...which is correct for here in the UK, but even if the system time was incorrect, it would still increment, but it's still showing 03:15 almost an hour later.
I am not saying system time is incorrect
I am saying
if you run script on same day all timestamps are the same.
If you run on a different day the timestamp has a different date and time.
It is a daily script after all.
Moving
# Set today's date on the current backup folder
touch $NR_DEST_PATH/daily.0
to here
# Back up
echo "Performing rsync backup ..."
rsync --archive --hard-links --delete --delete-excluded \
--exclude 'node_modules' --exclude 'data/node_modules' --exclude 'data/externalModules/node_modules' \
$NR_SOURCE_PATH/ $NR_DEST_PATH/daily.0
# Set today's date on the current backup folder
touch $NR_DEST_PATH/daily.0
# Validate return code
Sets the correct timestamp to the new backup folder, when backup on same day.
-rw-------. 1 u0_a102 u0_a102 0 Jan 4 11:23 .dtimestamp
drwx------. 6 u0_a102 u0_a102 4096 Jan 4 14:41 daily.0
drwx------. 6 u0_a102 u0_a102 4096 Jan 4 14:40 daily.1
drwx------. 6 u0_a102 u0_a102 4096 Jan 4 11:23 daily.2
drwx------. 6 u0_a102 u0_a102 4096 Jan 4 11:23 daily.3
drwx------. 6 u0_a102 u0_a102 4096 Jan 4 11:23 daily.4
drwx------. 6 u0_a102 u0_a102 4096 Jan 4 11:23 daily.5
drwx------. 6 u0_a102 u0_a102 4096 Jan 4 11:23 daily.6
drwx------. 3 u0_a102 u0_a102 4096 Jan 4 11:23 daily.7
drwx------. 2 u0_a102 u0_a102 4096 Jan 2 14:25 empty
This line does indeed set the timestamp on daily.0 to now:
drwxr-xr-x 5 pi pi 4096 Jan 4 13:06 /home/pi/nodered-backup/daily.0
# Set today's date on the current backup folder
touch $NR_DEST_PATH/daily.0
ls -ld $NR_DEST_PATH/daily.0 # My insertion
But (on my Pi), the rsync command changes it back:
Performing rsync backup ...
drwxr-xr-x 5 pi pi 4096 Jan 4 10:56 /home/pi/nodered-backup/daily.0
And 10:56 is the timestamp of the node-red folder (/home/pi/.node-red)
I'm not too familiar with rsync but it seems to be retaining the timestamp of the files it copies?
This is contrary to what man rsync says since the script is not using -t or --times flag.
Maybe a Pi does things differently...
@E1cid Thank you, your edit above resolves the issue
Now I've got proper timestamps against all of the backups.
drwxr-xr-x 5 pi pi 4096 Jan 4 16:08 daily.0
drwxr-xr-x 5 pi pi 4096 Jan 4 16:07 daily.1
drwxr-xr-x 5 pi pi 4096 Jan 4 16:06 daily.2
drwxr-xr-x 5 pi pi 4096 Jan 4 16:05 daily.3
drwxr-xr-x 5 pi pi 4096 Jan 4 16:04 daily.4
drwxr-xr-x 5 pi pi 4096 Jan 4 16:03 daily.5
drwxr-xr-x 5 pi pi 4096 Jan 4 16:02 daily.6
drwxr-xr-x 5 pi pi 4096 Jan 4 16:01 daily.7
So the full working example is;
#! /usr/bin/env bash
# Redirect stdout to syslog - show with: `sudo journalctl -t nodered-backup -n 100`
exec 1> >(logger -t nodered-backup -p local0.info)
# redirect stderr to syslog
exec 2> >(logger -t nodered-backup -p local0.err)
# --- SET THESE TO THE CORRECT LOCATIONS --- #
SOURCE_PATH=/home/pi/.node-red
DEST_PATH=/home/pi/backupDIR
# ------------------------------------------ #
STARTDATE=$(date +'%Y-%m-%d %T')
echo " "
echo "Starting daily backup of $SOURCE_PATH/ to $DEST_PATH/ ..."
echo "Rotating snapshots ..."
# Delete oldest daily backup
if [ -d $DEST_PATH/daily.7 ] ; then
echo " Deleting oldest daily backup $DEST_PATH/daily.7"
rm -rf $DEST_PATH/daily.7
fi
# Shift all other daily backups ahead one day
for OLD in 6 5 4 3 2 1 ; do
if [ -d $DEST_PATH/daily.$OLD ] ; then
NEW=$(($OLD+1))
echo " Moving $DEST_PATH/daily.$OLD to $DEST_PATH/daily.$NEW"
# Backup last date
# ISSUE: touch does not support options on synology (busybox) system
touch $DEST_PATH/.timestamp -r $DEST_PATH/daily.$OLD
mv $DEST_PATH/daily.$OLD $DEST_PATH/daily.$NEW
# Restore timestamp
touch $DEST_PATH/daily.$NEW -r $DEST_PATH/.timestamp
fi
done
# Copy hardlinked snapshot of level 0 to level 1 (before updating 0 via rsync)
if [ -d $DEST_PATH/daily.0 ] ; then
echo " Copying hardlinks from $DEST_PATH/daily.0 to $DEST_PATH/daily.1"
cp -al $DEST_PATH/daily.0 $DEST_PATH/daily.1
fi
echo "Finished rotating snapshots ..."
if ! [ -d $DEST_PATH/daily.0 ] ; then
mkdir -p $DEST_PATH/daily.0
fi
ENDDATE=$(date --iso-8601=s)
# Back up
echo "Performing rsync backup ..."
rsync --archive --hard-links --delete --delete-excluded \
--exclude 'node_modules' --exclude 'data/node_modules' \
$SOURCE_PATH/ $DEST_PATH/daily.0
# Set today's date on the current backup folder
touch $DEST_PATH/daily.0
# Validate return code
# 0 = no error,
# 24 is fine, happens when files are being touched during sync (logs etc)
# all other codes are fatal -- see man (1) rsync
if ! [ $? = 24 -o $? = 0 ] ; then
echo "Fatal: Node-RED daily backup finished with errors!"
#curl --insecure -I 'https://localhost:1880/nrnotify?type=backup&schedule=daily&result=fail'
mosquitto_pub -u username -P password -t services/$NR_SERVICE_NAME/backup/daily -m 'backup failed'
else
echo "Finished Node-RED daily backup, no errors."
#curl --insecure -I 'https://localhost:1880/nrnotify?type=backup&schedule=daily&result=success'
mosquitto_pub -u username -P password -t services/$NR_SERVICE_NAME/backup/daily -m 'successful backup'
fi
# Sync disks to make sure data is written to disk
sync
#EOF