This is a new node that makes a backup of an Influxdb database, using influxd backup.
It also adds some added value to that command.
The influxd backup command backs up one or all databases from a server to a folder and creates a set of .tar.gz files (plus manifest and meta files). The files are named in a format which includes the current time, YYYYMMDDTHHMMSSZ.*.tar.gz. Also there can be only one backup in a folder as the influxd restore command cannot cope with more than one in a folder. This is particularly unhelpful if the backup is part of a regular strategy that uses differential or deduplicated backups, such as Back in Time or Borg. The fact that each time a backup is performed it has to go to a new folder (unless the original files are deleted first) and the fact that the names are different each time can make the whole thing complicated. In addition a deduplicating backup app like Borg cannot make use of the fact that most of the backup files will be identical each time (for old data), not because the filenames change as Borg copes with that, but because the files are zipped the file contents are not the same each time (I guess the zip includes timestamps for the embedded files).
To ease the situation the node has a couple of features:
It has an option to remove old backup files from the folder specified.
It has an option to unzip the files (to .tar files), which makes files containing old data the same as at the last backup, and to rename the files to a predefined name. This means that the backup folder is largely unchanged each time a backup is performed, with just the addition of the most recent data. If it is required to restore from these files it is merely necessary to zip them up again before running the restore command (gzip *.tar).
If anyone has any interest in this node I would be most grateful if they could give it a go. There should be no chance of it damaging the influx data as it only uses influxd backup to access the server. In particular if anyone can check it on an alternative OS that would be good as I only have access to Ubuntu and Raspbian.
I have not published to npm yet so to install it run, from the .node-red folder npm install colinl/node-red-contrib-influxdb-backup
Version is currently 0.0.1. You will need to have git installed to do that. Once it has been published to npm then that will not be the case.
The readme and help are hopefully reasonably complete, but please also comment on any areas that need clarification there.
Documentation on the influx command can be found here. In particular see the section in there about configuring bind-address in influxdb.conf which is necessary to allow the backup command to work.
Which reminds me, I put that in the readme, but not in the help text I think, so that needs to go on the list.
Just tried to nseall it on my Mac (macOS 10.14.6) ad get an error
Paul$ npm install colinl/node-red-contrib-influxdb-backup
The authenticity of host 'github.com (140.82.112.3)' can't be established.5d5fa
RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8.
Are you sure you want to continue connecting (yes/no)? ⸨░░░░░░░░░░░░░░░░░░⸩ ⠧ ro⸨░░░░░░░░░░░░░░░░░░⸩ ⠧ rollbackFailedOptional: verb npm-session 10d4d78ef95d5fa
npm ERR! Error while executing:
npm ERR! /usr/local/bin/git ls-remote -h -t ssh://git@github.com/colinl/node-red-contrib-influxdb-backup.git
npm ERR!
npm ERR! Warning: Permanently added 'github.com,140.82.112.3' (RSA) to the list of known hosts.
npm ERR! git@github.com: Permission denied (publickey).
npm ERR! fatal: Could not read from remote repository.
npm ERR!
npm ERR! Please make sure you have the correct access rights
npm ERR! and the repository exists.
npm ERR!
npm ERR! exited with error code: 128
npm ERR! A complete log of this run can be found in:
npm ERR! /Users/Paul/.npm/_logs/2020-07-05T15_00_08_356Z-debug.log
[Update] just tried it on a Pi and get the same issue.
you should mention Git needs to be installed to gt it from your repository
why not use the same config that the influx nodes use to access the host, port and database? It would be node to select the same one that I've defined a already defined
(BUG?) does not work if Influx is not installed on the machine running the backup(??). In my case influx is on a remote machine and the influxdb nodes run fine but the backup fails with:
![Screen Shot 2020-07-05 at 12.27.49 PM|434x500]
(upload://8kaVnAcpLlHFF2Sh4rkTdyxprz3.png)
40 'Clear Folder' option - if it can only be 'true' or 'false' why are there extra options for the setting?
Good point, I have edited the post. That won't apply once it has been published to npm of course.
I am not sure about that. I think it would be necessary to install node-red-contrib-influxdb then I assume, and not everyone who uses influx use that. Also I have two influx servers backed up with one flow and pass the host across in a message. If it used a config node then I would have to use two separate backup nodes I think. So I don't know at the moment.
Yes, that is true. It is stated on the Help and in the readme. As far as I know there is no way round it. The influxd command needs to be available and I think that means that influxdb has to be installed. I don't know of a way to install that command on it's own. Perhaps I should explicitly check for that and produce a more helpful message though.
It has to be true or false, but since I allow other parameters to be passed in the message or via context or env var there didn't seem any reason not to allow that. If I were to not allow it you can bet that the first issue would be someone wanting to do it.
Sotty I neglected to read the help and readme, just started beating on it.
Interesting because I do not have influx installed on the Pi that does my inserts into the influx DB. It resides on another machine on my network (a Mac infact).
Does this mean influx must be running on every machine that uses the node? or just installed?
It is a bit confusing I agree. Normally the reads and writes are done over http so the client doesn't need anything special installed. Thus the influx server can be on one machine and node red can run on another and access the database via the node red nodes. The backup is done via a command (influxd) which runs on the client, and it (I assume) uses the normal http interface to fetch the data from the server. Even more confusingly if you just run influxd then that starts the server itself.
So no, influx does not need to be running on the client, it just needs to be available.
To be honest it is a bit of a dog's dinner, but it is all there is as far as I know.
By the way, if you were to test restoring (which my node doesn't do of course) I have just noticed in man influxd that it says you should stop the server before running a restore. I don't remember seeing that in the docs in the link though, so I will have to look into that.
Best way to test s/w I think, at least initially, if you RTFM and then only use it the way the book says you are much less likely to find the edge cases
That is odd, I get two out of the stderr output (I don't have much control of those, they are coming from the OS, there is the message on OP3 with the return code and one from the Catch node. I have just pushed a improvement which makes the error from the Catch node better (hopefully) if you could try that and then post all the messages you get (if it still seems unreasonable) then that would be good. To simulate it not installed I have moved away the influxd command, which should be good enough, but maybe not.
Did you see the note about setting up bind address on the server? You have to restart influxd after doing that.
ow to try and break it
