Backing up context data

[Hunter]

Hi all,

About 2 years ago, I posted the same issue;

Backing up data

Hello, I'n using NR to provide data to a large group of users (it was > 1000 but after I lost my data I'm back about 300 now). Each user has it's own settings, stored in aUsers object, stored als file : flow.set("aUsers", aUsers, "storeInFile"); Just to be sure, if I mess up anything, I created a change node which is injected once every day to backup my data. If necessary, I would be able to access a backed up copy of the user settings. So far, so good, the data gets copied into aUsersBackup …

Still, I'm facing the same issue and I'm wondering if there are any new thoughts on this?

Let me summarize my issue. My NodeRED feeds a Telegram system with about 3000 users. The user-data is stored in object as

flow.set("aUsers", aUsers, "storeInFile");

Once a day I would like to make a backup. In the beginning (years ago) I messed up this object once and I didn't have a backup. So I inject this every 24 hours :

const aUsers = flow.get("aUsers", "storeInFile"); flow.set("aUsers_backup", aUsers);

So far, so good. But the strange thing is, that this backup somehow is allways a live copy. Even if I stop injecting the backup code, the backup allways contains the same data. And, there comes the tricky part... If I delete my aUsers, the aUsers_backup also gets lost...?

I'm running NR in docker on NGINX.

Any suggestions maybe?

[marcus-j-davies]

Hi @Hunter,

Admittedly, I have not read the original thread.

But my hunch here, is referencing...

Javascript (like most) will pass-by-ref for objects/classes, whereas for primitive types, they are
pass-by-value

So what may be happening here, is your backup is a reference to the original (and not a copy)
so when the original changes, so does any 'reference' to it.

Try using RED.util.cloneMessage() to see if it makes a difference

const aUsers = flow.get("aUsers", "storeInFile");
flow.set("aUsers_backup", RED.util.cloneMessage(aUsers));

[TotallyInformation]

And the real issue is that your backup is not using the storeInFile type so it is only an in-memory copy which is, as stated, merely a link to the original data, not a copy.

---

Ideally, if you can take your Node-RED instance offline for a backup, that would ensure that the storeInFile context store was always in a valid state for the backup. Remembering that, by default, I think Node-RED only writes a file store to disk every 30 seconds.

I think that, if you really want a more robust store, it would be better to have a timed flow that manually writes the current context state to a file. Or even write a copy of inputs to a DB and restore from DB on startup. Database engines will have more robust backup processes.

[Hunter]

@marcus-j-davies

RED.util.cloneMessage()

Thanks for the tip mate, but this seems to be using some kind of reference as well

[Colin]

@Hunter

but this seems to be using some kind of reference as well

Show us what the code now is.

[kuema]

For such a use case, I'd go with a local SQLite database instead of using the file context. It is much more resilient regarding corruption, easy to integrate and to take backups.

[Hunter]

As stated.

At several points I save the user data as;

flow.set("aUsers", aUsers, "storeInFile");

As for the backup, I've tried 2 options;

#1
const aUsers = flow.get("aUsers", "storeInFile");
flow.set("aUsers_backup", aUsers);

#2
const aUsers = flow.get("aUsers", "storeInFile");
flow.set("aUsers_backup", RED.util.cloneMessage(aUsers));

Both ending up in the same behaviour as described.

[Colin]

@Hunter

const aUsers = flow.get("aUsers", "storeInFile");
flow.set("aUsers_backup", RED.util.cloneMessage(aUsers));

That should work. Can you post a small example flow showing it failing?

[Hunter]

Well, it doesn't

When I inject this code, the backup contains the same data, as expected. I use the node.warn() to show the number of users (size of object) which are both similar, as they should be.

When I register as new member, the main aUsers object increases by one, but when I run both Object.keys(aUsers).length and Object.keys(aUsers_backup).length they are both increased by one. So the backup is not an independant object, but locked to the main object.

[Colin]

[quote="Hunter, post:9, topic:82260"]
when I run both Object.keys(aUsers).length and Object.keys(aUsers_backup).length
[/quote

In that case the aUsers_backup variable you are using is not the one in created by
flow.set("aUsers_backup", RED.util.cloneMessage(aUsers));

@Colin

Can you post a small example flow showing it failing?

That is what we need.

[Hunter]

What do you mean by "failing"? There is no such failing message.
In fact, nothing fails. But the result is not as requested.

As long as I don't log any actions, there are no messages to be shown. So what is it you want to see?

There was no backup object present when I started this second topic. It was created by :

flow.set("aUsers_backup", RED.util.cloneMessage(aUsers));

There is no other code which can create this object...

[marcus-j-davies]

@Hunter

So what is it you want to see?

Just a small flow export reproducing the issue - of course using dummy data

[Colin]

Something like this possibly, but exhibiting the problem.

[{"id":"be7131e3813e4c1c","type":"inject","z":"bdd7be38.d3b55","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":120,"y":2580,"wires":[["c5f859709b461748"]]},{"id":"c5f859709b461748","type":"function","z":"bdd7be38.d3b55","name":"create context value and backup","func":"const obj = {test: 7}\nflow.set(\"obj\", obj)\nflow.set(\"obj_backup\", RED.util.cloneMessage(obj))\nreturn msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":340,"y":2580,"wires":[["a4503b334101f033"]]},{"id":"a4503b334101f033","type":"function","z":"bdd7be38.d3b55","name":"Fetch and test","func":"const obj = flow.get(\"obj\")\nconst objBackup = flow.get(\"obj_backup\")\nobj.newStuff = \"new stuff\"\nreturn [{payload: obj}, {payload: objBackup}];","outputs":2,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":580,"y":2580,"wires":[["7ed33b67ec465ce3"],["907c5650d0a2951b"]]},{"id":"7ed33b67ec465ce3","type":"debug","z":"bdd7be38.d3b55","name":"debug 103","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":770,"y":2540,"wires":[]},{"id":"907c5650d0a2951b","type":"debug","z":"bdd7be38.d3b55","name":"debug 104","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":770,"y":2600,"wires":[]}]

[Steve-Mcl]

If your data is important, I would personally recommend a database backup. Preferably on a separate computer to avoid the inevitable file corruption. It doesn't have to be anything fancy.

[TotallyInformation]

@Hunter

#1
const aUsers = flow.get("aUsers", "storeInFile");
flow.set("aUsers_backup", aUsers);

#2
const aUsers = flow.get("aUsers", "storeInFile");
flow.set("aUsers_backup", RED.util.cloneMessage(aUsers));

Both ending up in the same behaviour as described.

As previously stated, your flow.set is not using the file store and therefore is not a backup.