Deleting and Creating New Data Files

[edwardr]

I`m trying to find a way for data to be written to a file, so that when the file gets too big the file is either, stored, deleted and new file is put in its place?

So far Ive found a way to write to a file, however Im unsure how to make it so that Node Red at the end of the day deletes the data and starts new with a new file name etc - any ideas?

Ideally I would like it to input data into a dated file, that file is submitted to drop box by the end of the day, the file is then rewritten with the new date as the file name?

[JGKK]

You could either use the exec node with a rm command to delete the file if your under linux or use node-red-contrib-fs-ops for a more node internal version.
You could than use something node3-red-contrib-simpletime and a template node to construct a file name to be used by the file node on any given day.
Johannes

Edit
There is also a dropbox node

[edwardr]

Thanks JGKK, I`ll have a go now and get back to you with my progress

[TotallyInformation]

There is at least 1 contrib node that handles filing system actions so you could use those.

[JGKK]

is node-red-contrib-fs-ops that i linked above

[edwardr]

So using the dropbox node you can only submit or recieve from the drop box, ideally what I`d like to do is collect a days worth of data with a file name of the date- writing the data to the file is pretty easy, then I need to submit this to dropbox and then "refresh" the file with a new date as the file name and data rather than creating a new file ? is this possible?

[JGKK]

If you use the dropbox out node and leave it blank and pass in your file name as the msg.localFilename you can directly upload your file to dropbox.
Here is a little flow that gives you all the parts you need. Of course this is just the basic mechanism and youd have to set it up for your usecase.
Play with it a little and tell me if you have questions.
You will need the dropbox node + api token and the simpletime node installed for this flow to work:

[{"id":"69aeda7b.eecaa4","type":"dropbox out","z":"35572fe7.e9aef8","dropbox":"","filename":"","localFilename":"","name":"","x":1020,"y":400,"wires":[]},{"id":"66fa5bb4.b91864","type":"inject","z":"35572fe7.e9aef8","name":"write file","topic":"","payload":"this is a test","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":300,"y":400,"wires":[["5d655d42.916d44"]]},{"id":"71a45309.b492c4","type":"file","z":"35572fe7.e9aef8","name":"","filename":"","appendNewline":true,"createDir":false,"overwriteFile":"false","encoding":"none","x":610,"y":400,"wires":[[]]},{"id":"1c7c5c7d.082d44","type":"simpletime","z":"35572fe7.e9aef8","name":"","mydate":false,"myymd":true,"myyear":false,"mymonth":false,"mymonthn":false,"mydom":false,"mydoy":false,"myday":false,"myhourpm":false,"myhour":false,"mytime":false,"mytimes":false,"myminute":false,"myminutes":false,"mysecond":false,"mymillis":false,"myepoch":false,"myrawdate":false,"mypm":false,"x":470,"y":340,"wires":[["5f199dcd.8261c4"]]},{"id":"a11de1d.7d7c7a","type":"inject","z":"35572fe7.e9aef8","name":"Upload old / delete / save new name","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"00 00 * * *","once":false,"onceDelay":0.1,"x":210,"y":340,"wires":[["1c7c5c7d.082d44"]]},{"id":"65ced967.67379","type":"change","z":"35572fe7.e9aef8","name":"","rules":[{"t":"set","p":"localFilename","pt":"msg","to":"filename","tot":"flow"},{"t":"set","p":"filename","pt":"flow","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":820,"y":340,"wires":[["400d99e5.0e4668","69aeda7b.eecaa4"]]},{"id":"5f199dcd.8261c4","type":"template","z":"35572fe7.e9aef8","name":"","field":"payload","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"data-{{myymd}}.txt","output":"str","x":640,"y":340,"wires":[["65ced967.67379"]]},{"id":"400d99e5.0e4668","type":"delay","z":"35572fe7.e9aef8","name":"","pauseType":"delay","timeout":"5","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"x":1020,"y":340,"wires":[["1a89dc3.531d224"]]},{"id":"5d655d42.916d44","type":"change","z":"35572fe7.e9aef8","name":"filename","rules":[{"t":"set","p":"filename","pt":"msg","to":"filename","tot":"flow"}],"action":"","property":"","from":"","to":"","reg":false,"x":460,"y":400,"wires":[["71a45309.b492c4"]]},{"id":"c513502d.b7577","type":"exec","z":"35572fe7.e9aef8","command":"rm","addpay":true,"append":"","useSpawn":"false","timer":"","oldrc":false,"name":"","x":1370,"y":340,"wires":[[],["4845d10a.e64458"],["4845d10a.e64458"]]},{"id":"1a89dc3.531d224","type":"change","z":"35572fe7.e9aef8","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"localFilename","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":1200,"y":340,"wires":[["c513502d.b7577"]]},{"id":"4845d10a.e64458","type":"debug","z":"35572fe7.e9aef8","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":1530,"y":340,"wires":[]},{"id":"e4405484.85bf48","type":"inject","z":"35572fe7.e9aef8","name":"initialize filename","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":true,"onceDelay":0.1,"x":270,"y":280,"wires":[["4f51aa94.96f6d4"]]},{"id":"4f51aa94.96f6d4","type":"simpletime","z":"35572fe7.e9aef8","name":"","mydate":false,"myymd":true,"myyear":false,"mymonth":false,"mymonthn":false,"mydom":false,"mydoy":false,"myday":false,"myhourpm":false,"myhour":false,"mytime":false,"mytimes":false,"myminute":false,"myminutes":false,"mysecond":false,"mymillis":false,"myepoch":false,"myrawdate":false,"mypm":false,"x":470,"y":280,"wires":[["ed4dacff.7018a"]]},{"id":"87c6cd61.332e8","type":"change","z":"35572fe7.e9aef8","name":"","rules":[{"t":"set","p":"filename","pt":"flow","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":830,"y":280,"wires":[[]]},{"id":"ed4dacff.7018a","type":"template","z":"35572fe7.e9aef8","name":"","field":"payload","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"data-{{myymd}}.txt","output":"str","x":640,"y":280,"wires":[["87c6cd61.332e8"]]}]

Johannes

[edwardr]

Ive set up the dropbox node however it seems to only rewrite the file thats already in the dropbox, ideally I`d want a file for everyday , is there a way round that?

[TotallyInformation]

Add the date to the filename (or the day name if you want to rotate round 7 days for example).

[JGKK]

Thats exactly what the flow above shows. Everyday at midnight it creates a name that has the new date in it uploads the file from the last day and deletes it. The name for everyday that has the date is saved to flow.filename which is used by the file node to write the data to your filesystem.
So after a week you would have a file for everyday in your dropbox and only the file for the current day on your device. Look in detail at all the settings of the nodes in the middle part of the flow i posted. Especially the template and change nodes.

[edwardr]

Pardon me beiing a newbie! thank you for all your help - I`m using the flow but it seems to want to replace the file on dropbox and not add a file if that makes sense?

[JGKK]

Yes because the file name never changes as we are still in the same day. It will only create a new filename tomorrow as the date in ymd format is the unique part of the file name. If you want every file to be unique even if there are multiple file uploads on one day change the simpletime nodes to output the full date including also the time and change the property in the template nodes to use that instead or in addition to the ymd. This way you would have a file for every upload.

Edit
Here is the flow with the time added to the name so every time you upload it will be unique and no file be overwritten in the dropbox.

[{"id":"55cc8ae9.de1a4c","type":"dropbox out","z":"35572fe7.e9aef8","dropbox":"","filename":"","localFilename":"","name":"","x":1060,"y":340,"wires":[]},{"id":"a49a2a5a.8053f8","type":"inject","z":"35572fe7.e9aef8","name":"write file","topic":"","payload":"this is a test","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":340,"y":340,"wires":[["b08308e7.474da"]]},{"id":"8f0b4cb8.ebd598","type":"file","z":"35572fe7.e9aef8","name":"","filename":"","appendNewline":true,"createDir":false,"overwriteFile":"false","encoding":"none","x":650,"y":340,"wires":[[]]},{"id":"1c6c3b81.880a2c","type":"simpletime","z":"35572fe7.e9aef8","name":"","mydate":false,"myymd":true,"myyear":false,"mymonth":false,"mymonthn":false,"mydom":false,"mydoy":false,"myday":false,"myhourpm":false,"myhour":true,"mytime":false,"mytimes":false,"myminute":true,"myminutes":false,"mysecond":true,"mymillis":false,"myepoch":false,"myrawdate":false,"mypm":false,"x":510,"y":280,"wires":[["39e82b2d.0f45a4"]]},{"id":"f14c941b.9ca0e8","type":"inject","z":"35572fe7.e9aef8","name":"Upload old / delete / save new name","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"00 00 * * *","once":false,"onceDelay":0.1,"x":250,"y":280,"wires":[["1c6c3b81.880a2c"]]},{"id":"11d8b2bb.0c1cb5","type":"change","z":"35572fe7.e9aef8","name":"","rules":[{"t":"set","p":"localFilename","pt":"msg","to":"filename","tot":"flow"},{"t":"set","p":"filename","pt":"flow","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":860,"y":280,"wires":[["8388310a.4e9d78","55cc8ae9.de1a4c"]]},{"id":"39e82b2d.0f45a4","type":"template","z":"35572fe7.e9aef8","name":"","field":"payload","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"data-{{myymd}}-{{myhour}}{{myminute}}{{mysecond}}.txt","output":"str","x":680,"y":280,"wires":[["11d8b2bb.0c1cb5"]]},{"id":"8388310a.4e9d78","type":"delay","z":"35572fe7.e9aef8","name":"","pauseType":"delay","timeout":"5","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"x":1060,"y":280,"wires":[["25f53897.a513c8"]]},{"id":"b08308e7.474da","type":"change","z":"35572fe7.e9aef8","name":"filename","rules":[{"t":"set","p":"filename","pt":"msg","to":"filename","tot":"flow"}],"action":"","property":"","from":"","to":"","reg":false,"x":500,"y":340,"wires":[["8f0b4cb8.ebd598"]]},{"id":"c6c4a844.5cf938","type":"exec","z":"35572fe7.e9aef8","command":"rm","addpay":true,"append":"","useSpawn":"false","timer":"","oldrc":false,"name":"","x":1410,"y":280,"wires":[[],["7fc56a8e.2a02bc"],["7fc56a8e.2a02bc"]]},{"id":"25f53897.a513c8","type":"change","z":"35572fe7.e9aef8","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"localFilename","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":1240,"y":280,"wires":[["c6c4a844.5cf938"]]},{"id":"7fc56a8e.2a02bc","type":"debug","z":"35572fe7.e9aef8","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":1570,"y":280,"wires":[]},{"id":"b569651.819a698","type":"inject","z":"35572fe7.e9aef8","name":"initialize filename","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":true,"onceDelay":0.1,"x":310,"y":220,"wires":[["540e2e4e.846d4"]]},{"id":"540e2e4e.846d4","type":"simpletime","z":"35572fe7.e9aef8","name":"","mydate":false,"myymd":true,"myyear":false,"mymonth":false,"mymonthn":false,"mydom":false,"mydoy":false,"myday":false,"myhourpm":false,"myhour":true,"mytime":false,"mytimes":false,"myminute":true,"myminutes":false,"mysecond":true,"mymillis":false,"myepoch":false,"myrawdate":false,"mypm":false,"x":510,"y":220,"wires":[["e54f8114.48a8e"]]},{"id":"a328b9a5.3491e","type":"change","z":"35572fe7.e9aef8","name":"","rules":[{"t":"set","p":"filename","pt":"flow","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":870,"y":220,"wires":[[]]},{"id":"e54f8114.48a8e","type":"template","z":"35572fe7.e9aef8","name":"","field":"payload","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"data-{{myymd}}-{{myhour}}{{myminute}}{{mysecond}}.txt","output":"str","x":680,"y":220,"wires":[["a328b9a5.3491e"]]}]