Letting other nodes do stuf while one function node is working on a recursive search

I'm implementing a recursive search in a Function node in Node-RED.

The idea is to load directory listings in a loop and recursively search through all subfolders by calling the same function for each directory.

However, since the search can involve a large number of files and folders, I'd like to be able to abort it while it's running.

I initially tried using a flow variable for this. Another node would set the variable to true, and the recursive function would check its value on each call. If it was set to true, the function would return without recursing any further.

Unfortunately, this didn't work. Once the recursive search started, I couldn't interrupt it. It seemed that Node-RED was unable to update the flow variable while the Function node was busy executing the recursive search.

I managed to get around this by handling the recursion using separate nodes instead.

However, this approach makes it difficult to keep track of when the entire search is finished.

Is there a way to abort a running loop or recursive function inside a single Function node in Node-RED?

Hi @svefro

How are you executing your search, maybe a code example will help here?

it is important to note, Node JS (and therefore Node RED) is single threaded.
So if you are not making use of promises, you're choking the event loop - stopping your "escape" variable from actually getting updated, therefore witnessed by the loop.

I think the first part is correct, using a flow variable as a gate flag.

This is the issue I think, you've sent the main node.js loop into an impenetrable loop I suspect.

The node.js "way" to do this would, I think, be to create a worker thread, though even with that, you should make sure that you are using node's promise-based filing system methods and not synchronous ones. That should let you message the worker from the main thread in order to interrupt it.

Why not use recursive JS within your function node, using await for each readdir() call (to ensure it does not suffocate the JS engine), and checking the abort flag?
for example:

// In the node configuration, import fs.promises as "fs" from "node:fs", and path from "node:path"

await walkDir("path-to-tree-root");
msg.status = "OK";
return msg;

async function walkDir(dir)
{
	const list = await fs.readdir(dir, { withFileTypes: true });
	for (const entry of list)  {
		if (flow.get("abort") === true)  {
			msg.status = "Aborted";
			return msg;
		}
		const fullName = path.normalize(path.join(dir, entry.name));
		if (entry.isDirectory())  {
			await walkDir(fullName);	// Recurse into subdirectory
		}
		else if (entry.isFile())  {
          // process the file
		{
	}
}

One cancellation detail in the sample: return msg only exits the current walkDir call. The caller ignores that result, and the code after await walkDir(...) then overwrites the status:

Have walkDir return a boolean: false when aborted and true when finished. Propagate false from a child back to its caller, for example:

if (!(await walkDir(fullName))) return false;

Then set the final status once, at the outermost level:

const completed = await walkDir(rootDirectory);
msg.status = completed ? "OK" : "Aborted";
return msg;

This is a change to the sample's control flow, not a complete replacement for it. await fs.readdir(...) gives the event loop a chance to handle asynchronous I/O. If processing the files itself takes a long time, split that work into batches and yield with something like await new Promise(resolve => setTimeout(resolve, 0)) between batches. Merely awaiting Promise.resolve() repeatedly can still starve the incoming message that sets the abort flag.

i'm sorry i did not provide my code.

i was already using resursing inside my function
i did searches in subfilders with await Promise.all

const promises = subfolders_to_search.map((item) => searchRecurse(item));
await Promise.all(promises);

Solution: When i changed to just awaiting each subfolder i was able to abort like intended.
i set a 'global' variable when abort is detected so i do not need to propagate the abort back.

but i wonder why it works when doing:

await searchRecurse(item)

and not with:

const promises = subfolders_to_search.map((item) => searchRecurse(item));
await Promise.all(promises);

Any idea?

Complete working code here:

flow.set("SEARCH_ABORT", false)
if (context.get("SEARCHING") == true){
    node.warn("already SEARCHING")
    return null
}

context.set("SEARCHING", true)
msg.aborted = false

node.status({fill:"orange",shape:"ring",text:"Searching"});


let search_root = msg.payload
let search_for = msg.topic.toLowerCase()


let search_array = search_for.split("+")
node.warn(search_array)
let results = [] // not realy used for anything else than a count

function search(str,search_array){
    if (msg.aborted == true) return false
    let found = true
    for (let search_term of search_array){
        if (!str.includes(search_term) || search_term == "") {
            found = false
            break
        } 
    }
    return found

}

async function searchRecurse(path){
    //await timersPromises.scheduler.yield(); this has no effect
    //node.status({ fill: "orange", shape: "ring", text: "Searching: " + path}); // this uses network and cpu, disabled
    if (!path.endsWith("/")) {
        path += "/"
    }

    if (flow.get("SEARCH_ABORT") == true){
        msg.aborted = true
        //node.warn("Aborted")
        return // only returns out of this function, not parent functions
    }

    const entries = await fs.promises.readdir(path, { withFileTypes: true });

    let subfolders_to_search = []
    for (let entry of entries) {
        if (entry.isFile()) {

            let name = entry.name
            let found = search(name.toLowerCase(),search_array) // pathLowered + 

            if (found){
                let match = {
                    name: name,
                    path
                }
                results.push(match)
                node.send({ payload: match });
            }
            
        } else if (entry.isDirectory()) {
            let name = entry.name
            let found = search( name.toLowerCase(), search_array)
            if (found) {
                let match = {
                    name: name,
                    path,
                    type:"folder"
                }
                results.push(match)
                node.send({ payload: match });
            }

            // await each subfolder here. 
            await searchRecurse(path + entry.name)
            //no need to add to this 
            subfolders_to_search.push(path + entry.name)
            
        } else {
            
        }
    }

    // dont do this. makes it unable to abort
    //const promises = subfolders_to_search.map((item) => searchRecurse(item));
    //await Promise.all(promises);
}
await searchRecurse(search_root)

if (msg.aborted == true){
    node.warn("Aborted")
    //flow.set("SEARCH_ABORT", false) // do not set to false here. used after join node to stop results in flight
    node.status({ fill: "blue", shape: "ring", text: "Aborted. Found " + results.length  + " Files"});
    msg.payload = {
        type:"search_aborted",
        message: "Aborted. Found " + results.length  + " Files",
        count:results.length
    }
}else{
    node.status({ fill: "green", shape: "ring", text: "Done. Found " + results.length  + " Files"});
    msg.payload = {
        type:"search_done",
        message: "Done. Found " + results.length  + " Files",
        count:results.length
    }
}


context.set("SEARCHING", false)
return [null,msg];

by the way. results are fead trough a join node before going to a websocket, after this abort is checked to stop any results that might be in flight in the join node when aborting

Do you need this to work on different platforms?

It's hard to imagine getting close to the performance of OS commands for searching a directory tree.
On linux that might be file globbing or find / grep. No doubt other operating systems offer similar tools.

On a multi-core computer, running a search via exec, Node-red should not be impacted by the search?

I believe this is your issue. that promise.all can exhaust V8's thread pool and cause memory spikes when doing multi-folder searches.

So even though using a worker would certainly help, you would still need to work around that unless you use a different approach.

Are you pre-filtering for only specific file types, excluding irrelevant folders (.git for example) and avoiding a call to stat for each file? All of those can make things a lot more efficient.

Here is some AI suggested code, not tried it myself, it is just an example of some efficiencies. You could run several of these in parallel but not too many. A worker thread is still strongly recommended to run this in if doing big searches.

import { opendir } from 'node:fs/promises';
import { join } from 'node:path';

async function* walk(dir, skip = new Set(['node_modules', '.git'])) {
  for await (const entry of await opendir(dir)) {
    const full = join(dir, entry.name);
    if (entry.isDirectory()) {
      if (!skip.has(entry.name)) yield* walk(full, skip);
    } else {
      yield full;
    }
  }
}

Another approach, if wanting to stay within node.js would be to build your own index and search that instead. This is what I do with UIBUILDER's Markweb feature. The index is updated via a chokidar watch.

There is certainly an overhead in node.js. But the biggest difference would come, of course, by using an indexed OS search rather than an unindexed search. Clearly, searching an index will be a lot more performant.

Possibly a spawn with a streamed response?

yes using await Promise.all(promises); was faster but it most likley consumed the entire VM :stuck_out_tongue:

Thanks for all the pointers and suggestions.

Added filter for 'node_modules' and '.git' That was a good idea :+1:

Tested with an exec node and used find. But i get large strings that sometimes are split right in the middle of a line.
I can make it work. But for now i'll use the code i posted. it works fast enough. also many files are on a samba share so it is limited by that.

Well, I'm glad this thread came up because it made me realise that I've made the same mistake! :flushed_face:

Fortunately, the folder structures likely to be reviewed are somewhat smaller. But still, I've some work to do to improve both my watcher function and the build/rebuild of my internal index.

Another rabbit hole to go down. :exploding_head:

Of course! Thanks for the correction.

What are you exactly looking (searching) for files or directories ? Nothing beats find on a linux filesystem. Or locate - if it is being updated via some cronjob.

Find example;

find "/somepath" | grep -iE 'term1.*term2|term2.*term1'

You can use -type d and/or -type f if only searching for directory or filenames.
Nothing to abort either as it will be 20x faster than a readdir

Indeed.

Is that with the exec node in spawn or exec mode?
Personally I prefer the default exec mode. Never noticed an issue with spurious new lines.

Perhaps they aren’t using Linux or they need a cross platform solution. Or something more flexible.

Why not put such heavy tasks to an external Python script & benefit from multi threading or multiprocessing??? Cross platform also made simpler

Why? When Node.js can do all of that itself?