Backing up data

General
1

Hello,

I'n using NR to provide data to a large group of users (it was > 1000 but after I lost my data I'm back about 300 now). Each user has it's own settings, stored in aUsers object, stored als file :

flow.set("aUsers", aUsers, "storeInFile");

Just to be sure, if I mess up anything, I created a change node which is injected once every day to backup my data. If necessary, I would be able to access a backed up copy of the user settings.

So far, so good, the data gets copied into aUsersBackup

But then this day came, when I accidentally deleted all user data. So I wanted to use the backup, but it was also empty???

Asked users to re-register and the number of users (and settings) is increasing again. Monitored the backup this time and I noticed that the file size is equal at any time? So I doubted the change node, killed it and created a function in stead;

var aUsers = flow.get("aUsers", "storeInFile");
flow.set("aUsersBackup", aUsers, "storeInFile");

But still the same issue. So I disabled the trigger, so this function never gets triggered anymore.

Still the backup keeps the same size. Something seems to hardcopy it realtime.

Any suggestions maybe?

I just deleted the backup file and re-enabled the trigger towards the function node. Monitoring...

2

It may be worthwhile adding something to your function node, to log whenever the backup function is run, maybe;

var aUsers = flow.get("aUsers", "storeInFile");
flow.set("aUsersBackup", aUsers, "storeInFile");
node.log("Backup routine executed");

Then if you check the node-RED log, you will be able to see how frequently the routine is executed.
27 Nov 14:14:18 - [info] [function:5706867d93d8e57a] Backup routine executed

3

some suggestions
Make sure before saving the backup that it is not empty.
Maybe have the backup to a file, better still a new file everyday, maybe keeping the last 30 files just incase.

4

Have you changes settings.js to save the context data to storage?
https://nodered.org/docs/user-guide/context

5

Yes. I'm using context data to storage for a long while since I lost some data when restarting NodeRED in the past. After switching to storage this issue was solved. So settings should be OK.

6

I still did not figure out what is going wrong. Renamed the backup to something totally different (not containing the original name), tried to write the backup to memory, where the original is written to file.

No matter what I try, if a single inject to the backup function if performed, the backup starts to be a live copy. Whenever data is changed in the original context, the backup changes too.

Are there any more settings I can check, or a different way to copy/backup context?

Can anyone confirm that my way of copying should be able to behave at it should and that this is not a bug?

Just to clarify once more, I tried 2 things;

  • A change node which sets aUsersBackup to aUsers
  • A function node which gets the aUsers context and sets it to the backup aUsersBackup context

Both create a copy as they should, but then again; after a single inject the backup context always contains a live copy of the original context, making the backup not useful. When I accidentally delete (too much) data, there is no way the backup can help me :frowning:

7

Couple questions:

  1. what device is NR running on?
  2. how much storage and how much free storage does the device have?
  3. what version of NR and node.js? (you can get this from the startup log)
  4. why not store the user data in a database?
8

Sounds like a weird js reference issue. When you reference a variable, js generally creates a link rather than trying actually copy the data.

Maybe try putting a deep-clone between the get and the set, setting the clone, not the original.

Personally, my backups are done daily (x7), weekly (x4) and monthly (x12) using shell scripts and RSYNC and triggered using CRON. This is very reliable and efficient since the script only creates links if nothing has changed. I've posted the scripts before but happy to do so again if anyone wants them - though bear in mind that they are configured to back up local installs of Node-RED, not global ones. Easily adjusted though.

9

...or you could create a off-site backup routine by using Dropbox - node-RED Backup Flow (shameless plug :grin:)

10

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.