# \[ANNOUNCE\]node-red-cleanup-filesystem - request for testing

**URL:** <https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135>\
**Category:** Share Your Nodes\
**Created:** [21 May 2024 06:01 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135 "2024-05-21T06:01:31Z")\
**Posts on this page:** 20\
**Page:** 3

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [26 May 2024 23:38 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/41 "2024-05-26T23:38:58Z")

</div>

> [@Steve-Mcl](#):
>
> Not sure that is the right approach.
> 
> I would recommend a check prior to any operations that join the paths, resolves the joined paths, extract the base & test the resolved paths contains the base. If not, raise an error. Along those line anyhow.

Well, it is effective. 😀 It absolutely prevents escaping from the required root folder. Following the standard that you always validate user input as close to source as possible. Replacing `..` with an empty string and ensuring there is no leading `/` - or better still, prepending the absolute root folder to the start of the string - gives absolute certainty that an input cannot escape. And is pretty easy to do as well.

---

<div class="post-metadata">

**Author:** ![zenofmud](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/zenofmud/32/316_2.png) [@zenofmud](https://discourse.nodered.org/u/zenofmud)\
**Post date:** [27 May 2024 11:35 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/42 "2024-05-27T11:35:00Z")

</div>

Working much better with the latest version! Do you think it would be appropriate it you had two files with the same extension but one in lowercase and one in upper:

file1.txt  
file2.TXT

to delete them both by converting all file names and patterns to lowercase before processing?

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [27 May 2024 12:40 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/43 "2024-05-27T12:40:33Z")

</div>

no, most file systems are case-sensitive by design and assumptions should not be made IMO.

You can however use glob patterns to achieve this. e.g `*.{log,LOG}`  
Try it out here: [Glob Tool | DigitalOcean](https://www.digitalocean.com/community/tools/glob?comments=true&glob=sub%2F%2A.%7Blog%2CLOG%7D&matches=false&tests=flowfuse-broker-parsed.txt&tests=flowfuse-broker.txt&tests=log%20-%20Copy.log&tests=sub%2Fflowfuse-broker-parsed.txt.log&tests=sub%2Fflowfuse-broker-parsed.txt.LOG)

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [27 May 2024 12:44 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/44 "2024-05-27T12:44:41Z")

</div>

Bart, gave it a quick go, something funky going on

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/e/a/ea484dfc3f1d2d13baf406731726f707e2e68c22.png)

This is what I expected: [Glob Tool | DigitalOcean](https://www.digitalocean.com/community/tools/glob?comments=true&glob=%2A%2A%2F%2A.log&matches=false&options=dot%3Atrue&tests=%2Fin-temp-log-1.txt&tests=%2Fin-temp-log-2.log&tests=%2Fin-temp-log-3.txt&tests=%2Fsub%2Fin-temp-log-sub-1.log&tests=%2Fsub%2Fin-temp-log-sub-2.txt)

* * *

EDIT

This may be partly down to how I entered paths and being on a windows system. e.g. when I set the base path to use `x\y\z` vs `x/y/z` format I get different results.

I _think_ what is happening is there is no path normalisation going on. e.g. to be platform agnostic, I typically convert all paths to posix format and resolve them to ensure sane paths. Also (i didnt check, you may already be doing this) but I also use nodes `path.join` to ensure no double slashes or missing slashes in the final joined paths.

---

<div class="post-metadata">

**Author:** ![jbudd](https://avatars.discourse-cdn.com/v4/letter/j/5f8ce5/32.png) [@jbudd](https://discourse.nodered.org/u/jbudd)\
**Post date:** [27 May 2024 13:55 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/45 "2024-05-27T13:55:22Z")

</div>

A few thoughts that struck me:

> [@BartButenaers](#):
>
> `npm install bartbutenaers/node-red-cleanup-filesystem`

The node-name File Retention Manager is not very helpful.  
I found it a bit confusing that after installing cleanup-filesystem, searching the palette for cleanup or filesystem did not show it.

Someone in need of a node to delete unwanted files rather than the command line is surely going to be intimidated by having to choose between glob and regex, never mind constructing these filters?

A pattern is not mandatory but age is. Why?

The node seems to ignore the age setting for empty directories:  
mkdir /home/pi/stuff/brandnew  
Immediately run the flow with age set to 5 min  
report.folders includes brandnew

```auto
[
"/home/pi/stuff/brandnew",
"/home/pi/stuff/directorywithfiles/emptydirectory",
"/home/pi/stuff/emptydirectory"
]

```

For setup purposes it might be handy if the report array included the modification time for each file.

The debug pane output would be more compact if the base folder was not shown on each line.

Edit - another niggle:  
The output in msg.payload should be msg.payload.result. I've spent half an hour trying to find any glob pattern that works, only to realise that msg.payload.deletedFiles is empty because the dry run flag is set, not because the pattern is broken.  
I know, it is _deleted_Files but still it confused me.

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [27 May 2024 14:13 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/46 "2024-05-27T14:13:12Z")

</div>

> [@jbudd](#):
>
> The node-name File Retention Manager is not very helpful.

Agree! Even though I know what it is (due to experience and being a programmer), I dont hink it is _ideal_ for low-code terms.

> [@jbudd](#):
>
> Someone in need of a node to delete unwanted files rather than the command line is surely going to be intimidated by having to choose between glob and regex, never mind constructing these filters?

Kinda on side here. I am _usually_ of the opinion "choice is good" but I am not sure regex is really warranted in this case (tho I am happy Bart defaulted to `glob`)

  

@BartButenaers curious as to why you added it to this "filesystem" category. Do you have other file type nodes using that category? As you can see, your node is all alone in there 😢  
Perhaps the `storage` category (alongside the file read/write nodes) could be a better fit ?(NB: the `storage` category is internationalised and therefore more discoverable for foreign speakers) :

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/d/0/d0a611ae359543b9643ac00c7624299d1e97dcb7.png)

---

<div class="post-metadata">

**Author:** ![jbudd](https://avatars.discourse-cdn.com/v4/letter/j/5f8ce5/32.png) [@jbudd](https://discourse.nodered.org/u/jbudd)\
**Post date:** [27 May 2024 15:12 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/47 "2024-05-27T15:12:34Z")

</div>

Using the glob pattern `**/*junk*` offers these files for deletion (files with a problem underlined)  
 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/d/3/d38def9401db5c984da72224faae2b666e46d8e7.png)

1. From a Linux perspective I would not expect `.junk1` to be discovered by this glob. Compare with  
 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/1/7/17caae103b9067f4043843bc2341530cc1e87e27.png)  
Edit - But I see that `find` also shows dotted filenames.  
 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/4/8/485c7a6b3e643c4bc592e2078aa8c82194b59b34.png)

2. Using the glob pattern `**/?junk*` however, which should discover `sjunk1` does not do so.  
 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/8/4/8428b44185121a07ba67ff67558d049b64fa52d5.png)  
cf  
 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/a/7/a70ca83a5f029e3d4071c425718059b87f7f715f.png)

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [27 May 2024 17:02 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/48 "2024-05-27T17:02:07Z")

</div>

> [@jbudd](#):
>
> From a Linux perspective I would not expect `.junk1` to be discovered by this glob.

Try `ls -a **/*junk*`  
By default `ls` does not show hidden files.

---

<div class="post-metadata">

**Author:** ![jbudd](https://avatars.discourse-cdn.com/v4/letter/j/5f8ce5/32.png) [@jbudd](https://discourse.nodered.org/u/jbudd)\
**Post date:** [27 May 2024 17:08 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/49 "2024-05-27T17:08:54Z")

</div>

Hmm. Thanks Colin.

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [27 May 2024 17:33 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/50 "2024-05-27T17:33:56Z")

</div>

> [@Steve-Mcl](#):
>
> I also use nodes `path.join` to ensure

also path.normalise is useful to resolve those that you can then check are still valid (or in this case still start with the base dir.)

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [27 May 2024 21:32 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/51 "2024-05-27T21:32:47Z")

</div>

> [@Steve-Mcl](#):
>
> I _think_ what is happening is there is no path normalisation going on

Hey Steve,  
Thanks for the tip! I used already `path.join` Not used the path resolving before.  
I have added it to Github. Hopefully that solves it...

> [@jbudd](#):
>
> I found it a bit confusing that after installing cleanup-filesystem, searching the palette for cleanup or filesystem did not show it.

I think this is because during this discussion I changed the node name to \*node-red-file-retention-manager'. So when I enter _"file"_ or _"retention"_ I see it popping up:

![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/9/4/94ac7c2e06100531e713baefc6e6d5d3b0aefad1.png)  
 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/1/9/19d8fe1a324187ae62e484e9e8d12fb2e758d1c4.png)

> [@jbudd](#):
>
> The node-name File Retention Manager is not very helpful.

That might be the case. Not sure. For me it is very clear, but might be unclear for others. No idea. That is why I asked some renaming proposals above...

> [@jbudd](#):
>
> intimidated by having to choose between glob and regex

Personally I like regex much more, but I have added glob because I assumed from the above discussion that it is more widely used for file related searches. I will add some extra info in the documentation to make the distinction a bit more clear, and some example patterns to get them started. I have made "glob" the default option, but I am going to keep regex because I like it more. And I am the most important customer for this node 😉

> [@jbudd](#):
>
> A pattern is not mandatory but age is. Why?

That was not the intention, because I added a [check](https://github.com/bartbutenaers/node-red-file-retention-manager/blob/main/cleanup_fs.js#L81) to make sure a pattern is defined. Need to work tomorrow evening, so will try to fix it in a couple of days from now.

> [@jbudd](#):
>
> The node seems to ignore the age setting for empty directories

Hmm that might indeed be that case. My use case for this node was to remove the video footage from my Reolink doorbell. So the focus is on removing files, and then check whether to remove the folder that had become empty. Will need to have a look at that also.

> [@jbudd](#):
>
> For setup purposes it might be handy if the report array included the modification time for each file.

Yes indeed that might be usefull. Had not thought about that, because for Reolink the directory structure already contains the year & month & day, so I had already all that info in my report indirectly. Need to have a look at that also...

> [@jbudd](#):
>
> The debug pane output would be more compact if the base folder was not shown on each line.

Yes true. On the other hand if you have multiple instances of this node (each with their own base folder), it is not clear anymore which files have been removed...

> [@jbudd](#):
>
> only to realise that msg.payload.deletedFiles is empty because the dry run flag is set, not because the pattern is broken.

Yes I had added that to the documentation, because @zenofmud was also confused in the beginning. I have already considered to calculate _deletedFiles_ and _deletedFolders_ in case of a dry-run, but I didn't do that for two reasons:

- Now it is clear to me that the dry-run did not delete anything.
- Otherwise it would be some kind of redundant data in the output msg, because it contains the same info as the _ **length** _ of the _msg.payload.report.files_ and _msg.payload.report.folders_ arrays.

Perhaps I need to explain it more clearly both in the info panel and readme page.

> [@Steve-Mcl](#):
>
> @BartButenaers curious as to why you added it to this "filesystem" category. Do you have other file type nodes using that category? As you can see, your node is all alone in there 😢

My node has not be complaining about if feeling lonely...  
Choosing a category is always about creative. There is no global list of categories to choose from, and I don't have time to start searching for all related nodes across the web to figure out which category they have used. Moreover I don't always like categories that other developers have choosen. Moreover they oftern use _"function"_ because they have copied the code from another node and forgot to change it... So I have choosen "filesystem" simply because I liked it.

But yes I can change it to "storage"...

> [@Colin](#):
>
> Try `ls -a **/*junk*`  
> By default `ls` does not show hidden files.

Thanks for the assitance @Colin !

---

<div class="post-metadata">

**Author:** ![Paul-Reed](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/paul-reed/32/66906_2.png) [@Paul-Reed](https://discourse.nodered.org/u/Paul-Reed)\
**Post date:** [27 May 2024 22:10 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/52 "2024-05-27T22:10:28Z")

</div>

@BartButenaers - **As a test** , I let your node loose on my music folder earlier, which contained a large number of sub-directories and .mp3 files, where the names for both contained spaces, such as `Alter Bridge` and tracks `11.Clear Horizon.mp3`.  
I successfully deleted a number of files & folders using `glob` - (music that I uploaded for my grandson 8 years ago!) which all worked fine!

All of the nursery rhymes are now gone 😁

---

<div class="post-metadata">

**Author:** ![jbudd](https://avatars.discourse-cdn.com/v4/letter/j/5f8ce5/32.png) [@jbudd](https://discourse.nodered.org/u/jbudd)\
**Post date:** [27 May 2024 23:52 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/53 "2024-05-27T23:52:26Z")

</div>

You seem to be applying the regex pattern to the full pathname of each file, thus regex  
`.{15}junk.*` finds 6 of my test files to delete  
 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/f/1/f17f10a1ad23d562a1adf2ba17900878f6f8dbbd.png)  
and `^junk.*` finds none.

This seems a little surprising, especially so if at some future time "base folder" can be passed to the node as a message property and is thus of indeterminate length.

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [28 May 2024 05:25 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/54 "2024-05-28T05:25:38Z")

</div>

> [@jbudd](#):
>
> This seems a little surprising

Yes indeed you are right. Will need to change it. Thanks for testing!

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [6 June 2024 18:43 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/55 "2024-06-06T18:43:02Z")

</div>

Hi guys,

Did quite some refactoring, and hopefully most of the feedback is now implemented in one or another way. The [readme](https://github.com/bartbutenaers/node-red-file-retention-manager/tree/main) page is up to date, in case anybody wants to test this new version...

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [7 June 2024 05:55 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/56 "2024-06-07T05:55:53Z")

</div>

Got some feedback that the current name _"node-red-file-retention-manager"_ doesn't clearly explain what the node does. So I am willing to rename it one more time, to explain more clear that it is used only to remove files (and optionally folders) that have exceeded a specified age.

So let's do a poll:

_Poll ([view on site](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/56))_

You can select 2 options maximum, and I will close the poll in a couple of days.  
Thanks!!

---

<div class="post-metadata">

**Author:** ![kevinGodell](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/kevingodell/32/27040_2.png) [@kevinGodell](https://discourse.nodered.org/u/kevinGodell)\
**Post date:** [7 June 2024 14:18 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/57 "2024-06-07T14:18:00Z")

</div>

- node-red-rm (to keep the name simple so that the experts will find it)
- node-red-delete-file-by-age (chatgpt helped with this one)

The name with `eraser` makes me think of the contents of the file being erased as opposed to the file being unlinked. The name with `retention` leads me to think that it focuses on retaining more than it does on removing, such as moving the files to another location for a more permanent long-term storage.

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [7 June 2024 15:48 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/58 "2024-06-07T15:48:53Z")

</div>

How about. node-red-delete-aged-files

---

<div class="post-metadata">

**Author:** ![Frida](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/frida/32/68250_2.png) [@Frida](https://discourse.nodered.org/u/Frida)\
**Post date:** [7 June 2024 17:13 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/59 "2024-06-07T17:13:14Z")

</div>

When I have worked with retention files in a database, what should I look for in NodeRed?

```auto
The InfluxDB retention enforcement service checks for and removes
data with timestamps beyond the defined retention period of the
bucket the data is stored in. This service is designed to automatically
delete "expired" data and optimize disk usage without any user
intervention.

```

---

<div class="post-metadata">

**Author:** ![Paul-Reed](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/paul-reed/32/66906_2.png) [@Paul-Reed](https://discourse.nodered.org/u/Paul-Reed)\
**Post date:** [7 June 2024 18:32 UTC](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135/60 "2024-06-07T18:32:39Z")

</div>

What about `node-red-file-blaster`

Or... `node-red-file-remover`?

[Previous page](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135.md?page=2)

[Next page](https://discourse.nodered.org/t/announce-node-red-cleanup-filesystem-request-for-testing/88135.md?page=4)
