# \[ANNOUNCE\]node-red-cleanup-filesystem - request for testing
**URL:** https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135
**Category:** Share Your Nodes
**Created:** [21 May 2024 06:01 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135 "2024-05-21T06:01:31Z")
**Posts on this page:** 20
**Page:** 2
### Author:  [@gerry](https://discourse.nodered.org/u/gerry)
#### Post date: [24 May 2024 21:14 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/21 "2024-05-24T21:14:09Z")
Long before somebody gets close enough to clone a key I figure the dog will get them
---
### Author:  [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)
#### Post date: [26 May 2024 07:17 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/22 "2024-05-26T07:17:02Z")
Thanks for the feedback!!
I have implemented the changes from @Steve-Mcl to support (by default) glob patterns and to use a list instead of a `;` separated string. Which were obvious feature requests...
Now I am going to start finalizing this node, because it took me more time as expected already...
Would be nice if @zenofmud could repeat his tests on this new implementation 🙏
There is one thing I don't like about this node, and that is the name. A name like _node-red-contrib-cleanup-filesystem_ creates too much expectations about being a general purpose filesystem cleanup node. Which is _ **not** _ the case. It simply allows you to cleanup files (and folders) in a subtree below some base folder, if those files have exceeded some retention period (i.e. age). Which is quite convenient to cleanup video footage from my IP camera's...
So I would like to rename it, but not sure what name would be best:
- node-red-cleanup-aged-files
- node-red-file-retention-manager
- ...
Anybody an idea for a good name?
Thanks!!
Bart
---
### Author:  [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)
#### Post date: [26 May 2024 08:11 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/23 "2024-05-26T08:11:00Z")
> [@BartButenaers](#):
>
> Anybody an idea for a good name
Scoped would be a sensible start 😉
node-red-directory-cleanse
node-red-file-purge
node-red-file-eraser
node-red-file-exterminator
node-red-file-remover
node-red-file-cleaner
However, if you suspect additional features like rotation or zip & delete source, file moves, bulk renames etc, something more generic would be sensible.
node-red-file-operations
node-red-fs-toolkit
node-red-filesystem-tools
node-red-file-manager
Any good?
---
### Author:  [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)
#### Post date: [26 May 2024 09:40 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/24 "2024-05-26T09:40:55Z")
> [@Steve-Mcl](#):
>
> However, if you suspect additional features like rotation or zip & delete source, file moves, bulk renames etc, something more generic would be sensible.
Yeah that is what I meant that this node is not a general purpose filesystem cleanup node. Just need now something that can fit in my free time. So I am going to keep it simple this time, otherwise I will never be able to complete my own home automation...
> [@Steve-Mcl](#):
>
> Scoped would be a sensible start
Have you been talking to chatgpt 🤣
I did but my keywords were perhaps not very good, because I was not really impressed with his creativity this time...
Things _ **node-red-file-purge** _ doesn't really tell WHY the files have becoming obsolete. In this case it is only because the files are too old, or have expired, ... Something in that area. But my creativity in english is a bit too limited to come something for that.
---
### Author:  [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)
#### Post date: [26 May 2024 09:53 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/25 "2024-05-26T09:53:25Z")
> [@BartButenaers](#):
>
> > [@Steve-Mcl](#):
> >
> > Scoped would be a sensible start
>
> Have you been talking to chatgpt
Are you saying I sound like a computer 😅
I should have been more explicit. I meant `@scope/node-name` 😉
I think `@scope/filesystem-tools` is enough to convey filesystem based package and that it provides "tools" for the user.
`@scope/fs-toolkit` is a bit more cryptic (the fs part may not be obvious for non programmers) but I still like it.
I also like this: `@scope/file-operations`
I don't like `file-manager` (or similar) as it conveys something like a windows file manager (i.e. something with a UI)
---
### Author:  [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)
#### Post date: [26 May 2024 10:12 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/26 "2024-05-26T10:12:07Z")
> [@Steve-Mcl](#):
>
> I meant `@scope/node-name`
Ah yes. I use scopes indeed for my new nodes. And no "contrib" stuff anymore...
---
### Author:  [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)
#### Post date: [26 May 2024 12:03 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/27 "2024-05-26T12:03:34Z")
> [@BartButenaers](#):
>
> node-red-file-retention-manager
👍 I think that sums it up nicely.
---
### Author:  [@zenofmud](https://discourse.nodered.org/u/zenofmud)
#### Post date: [26 May 2024 13:18 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/28 "2024-05-26T13:18:08Z")
@BartButenaers Just tried to install the new version and I get this
```auto
paul@PaulsM1 .node-red % npm install bartbutenaers/node-red-cleanup-filesystem
npm notice
npm notice New major version of npm available! 8.19.2 -> 10.8.0
npm notice Changelog: https://github.com/npm/cli/releases/tag/v10.8.0
npm notice Run npm install -g npm@10.8.0 to update!
npm notice
npm ERR! code EJSONPARSE
npm ERR! path /Users/paul/.npm/_cacache/tmp/git-clone45Nwz9/package.json
npm ERR! JSON.parse Unexpected token ":" (0x3A) in JSON at position 408 while parsing near "... \"glob-to-regexp\": \"^0.4.1\"\n ],\n ..."
npm ERR! JSON.parse Failed to parse JSON data.
npm ERR! JSON.parse Note: package.json must be actual JSON, not just JavaScript.
npm ERR! A complete log of this run can be found in:
npm ERR! /Users/paul/.npm/_logs/2024-05-26T13_11_38_920Z-debug-0.log
```
Thoughts? (this is on my Mac mini M1 using Ventura v13.16.4)
UPDATE: I copied the package.json file into a JSON processor I have and see this:
---
### Author:  [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)
#### Post date: [26 May 2024 13:54 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/29 "2024-05-26T13:54:22Z")
> [@zenofmud](#):
>
> Thoughts?
Yes. Pebcak issue. Apologies!!
BTW I have renamed everything, and node is now called [node-red-file-retention-manager](https://github.com/bartbutenaers/node-red-file-retention-manager/blob/main/README.md)
---
### Author:  [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)
#### Post date: [26 May 2024 14:29 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/30 "2024-05-26T14:29:54Z")
~~bah, "retention-manager" sounds like something that manages a service or application that does "retention"!~~
~~I would never think to search for that when looking for something to do with files!~~
Ignore me - its fine as is 🙂
---
### Author:  [@Paul-Reed](https://discourse.nodered.org/u/Paul-Reed)
#### Post date: [26 May 2024 15:03 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/31 "2024-05-26T15:03:20Z")
In the UK, a Retention Manager usually refers to a managerial position held by someone tasked with retaining customer loyalty.
For example, people wanting to cancel their broadband service usually get put through to the Retention department, which is managed by a .... 😉
---
### Author:  [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)
#### Post date: [26 May 2024 15:08 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/32 "2024-05-26T15:08:42Z")
---
### Author:  [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)
#### Post date: [26 May 2024 16:32 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/33 "2024-05-26T16:32:37Z")
> [@Steve-Mcl](#):
>
> Ignore me - its fine as is
Oh what a relieve. That is good news.
I wanted to start the development of a new node, that allowed me to rename other nodes. Then I could rename my nodes e.g. every 5 seconds, without any human intervention 😋
The word _"retention"_ sounded immediately familiar to me, because we use Kafka at work. And Kafka uses a "retention period" to specify how long the messages need to be remembered on a topic...
---
### Author:  [@zenofmud](https://discourse.nodered.org/u/zenofmud)
#### Post date: [26 May 2024 16:33 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/34 "2024-05-26T16:33:59Z")
Crashed NR:
```auto
26 May 12:31:26 - [red] Uncaught Exception:
26 May 12:31:26 - [error] TypeError: Cannot read properties of undefined (reading 'length')
at fileRetentionManager._inputCallback (/Users/paul/.node-red/node_modules/@bartbutenaers/node-red-file-retention-manager/cleanup_fs.js:61:79)
at /usr/local/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:214:26
at Object.trigger (/usr/local/lib/node_modules/node-red/node_modules/@node-red/util/lib/hooks.js:166:13)
at Node._emitInput (/usr/local/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:206:11)
at Node.emit (/usr/local/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:190:25)
at Node.receive (/usr/local/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:499:10)
at deliverMessageToDestination (/usr/local/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/flows/Flow.js:803:40)
at Immediate. (/usr/local/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/flows/Flow.js:819:21)
at process.processImmediate (node:internal/timers:471:21)
```
here is the flow:
```auto
[{"id":"e8128048cf12e796","type":"inject","z":"f82aec0811679377","name":".PNG","props":[{"p":"payload.fileNamePattern","v":".PNG","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":270,"y":560,"wires":[["40f5d29127ba14e7"]]},{"id":"9e7deb562e878bc6","type":"debug","z":"f82aec0811679377","name":"debug 3437","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":830,"y":560,"wires":[]},{"id":"40f5d29127ba14e7","type":"file-retention-manager","z":"f82aec0811679377","name":"","baseFolder":"/users/paul/bart_test","patternType":"glob","age":"1","ageUnit":"minutes","removeEmptyFolders":false,"dryRun":false,"report":true,"patterns":[],"x":540,"y":560,"wires":[["9e7deb562e878bc6"]]}]
```
Note I had not changed the `payload.fileNamePattern` in the inject node. However after changing the `inject` node to use `payload.patternType` it still crashed NR.
---
### Author:  [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)
#### Post date: [26 May 2024 16:49 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/35 "2024-05-26T16:49:09Z")
> [@zenofmud](#):
>
> Crashed NR:
Should be fixed now on Github.
Nice catch!! But not catched by the code...
---
### Author:  [@zenofmud](https://discourse.nodered.org/u/zenofmud)
#### Post date: [26 May 2024 18:51 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/36 "2024-05-26T18:51:40Z")
Ok, I can't figure out how to use glob and if I change to REGEX and run pointing to a folder with three items, this is what I see:
the error seems to point to line 137 in cleanup\_fs.js::
if (fileAge \> ageInSeconds && patterns.some(pattern =\> pattern.test(filePath))) {
---
### Author:  [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)
#### Post date: [26 May 2024 18:59 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/37 "2024-05-26T18:59:19Z")
Bart, I'm sure you know but glob patterns can be `/../../xxx` meaning a user could accidentally delete stuff from a directory above the base folder - do you sanity check paths do not resolve outside of base folder?
---
### Author:  [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)
#### Post date: [26 May 2024 19:57 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/38 "2024-05-26T19:57:50Z")
> [@BartButenaers](#):
>
> "retention period"
Retention period is also an Information Governance term related to privacy and things like GDPR. For personal information, you are only permitted to retain things for set periods. So this is a very familiar term.
> [@Steve-Mcl](#):
>
> Bart, I'm sure you know but glob patterns can be `/../../xxx` meaning a user could accidentally delete stuff from a directory above the base folder - do you sanity check paths do not resolve outside of base folder?
Yes, this is the first thing to do. Remove all text containing `..`. Similarly, you may need to prevent absolute references as well (leading `/`). Also remove everything before the LAST `:` found. Which prevents the use of networked file systems as well as preventing the use of different drives on Windows.
---
### Author:  [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)
#### Post date: [26 May 2024 20:46 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/39 "2024-05-26T20:46:01Z")
> [@TotallyInformation](#):
>
> Remove all text containing `..`
Not sure that is the right approach.
I would recommend a check prior to any operations that join the paths, resolves the joined paths, extract the base & test the resolved paths contains the base. If not, raise an error. Along those line anyhow.
---
### Author:  [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)
#### Post date: [26 May 2024 20:58 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/40 "2024-05-26T20:58:06Z")
> [@zenofmud](#):
>
> the error seems to point to line 137 in cleanup\_fs.js::
Should be fixed now on Github. Was indeed not working for most regex expressions.
> [@zenofmud](#):
>
> I can't figure out how to use glob
Well I have to admit that I hadn't used it also not before. Do you want to find all png files like in your screenshot? Just ask chatgpt 😉
The pattern seems to work...
> [@Steve-Mcl](#):
>
> meaning a user could accidentally delete stuff from a directory above the base folde
Normally not. The node does a traversel of all folders starting from the base folder down. And then each folder is matched with the pattern. So not only a traversal of the folders specified in the patterns.
By doing it that way I can use the patterns to specify subfolders and filenames or extensions. Just what I need to cleanup the video footage and snapshot images from my camera's. For example a ReoLink doorbell generates a lot of subfolders ({year}/{month}/{day}), which I can now easily clean up.
Nothing more, nothing less. As I mentioned before, it is not a full blown filesystem cleanup node. I didn't design it to process a large amount of files or directories, just something for my own use case that I share with others on this blue planet 😉
> [@TotallyInformation](#):
>
> So this is a very familiar term.
Nice to get a confirmation from a native speaking English human being...
[Previous page](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135.md?page=1)
[Next page](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135.md?page=3)