Select files within a date range and merge them into one file

General
8

Here is another way to do it :grinning:

Using just a function node.
image
You can use fs instead of fs-extra if you like but then you will end up with a more complex first line of the for block. fx-extra is used by Node-RED by the way. I'm using fast-glob here instead of using the fs-ops node.

// Use Fast-Glob to get the list of files
const entries = await fg(['*/csv/*.csv'])
msg.entries = entries

// get the files
msg.files = []
let fline
for (const fname of entries) {
    let f = await fs.readFile(fname, 'utf8')
    f = f.split('\n')
    fline = f.shift() // remove the first line
    f = f.join('\n')

    msg.files.push( f )
}

// Add the first line back and join all the headerless files together
msg.payload = `${fline}\n${msg.files.join('')}`

return msg

It uses a number of interesting modern JavaScript constructs so may be a useful learning exercise as well.

Obviously, this approach as with the others given has a problem. You can run out of memory. You can improve things by not keeping the individual files on the msg as well as the final output but you can still easily run out of memory if handling very large files.

So for large file handling, it is better to use a streaming approach whereby you stream in the files and stream them straight out to the output file.

The other limitation of this approach is that it only works if the columns are exactly the same for each input file.

Incidentally, both Python (via libraries) and PowerShell (no need for libraries) are better at CSV handling than JavaScript is.

Oh, and of course, you should look on npmjs.org for libraries that will do CSV merge directly - there are plenty.