Attempting to create a .txt data reader/organizer

General
1

Hi all,

I'm a new user of Node-Red and currently attempting to build a specific data reader/organizer, please excuse my ignorance if something like this has been answered before, I did a quick search around the forums and flows however nothing turns up quite similar.

I want to write function that would allow me to read all files in a specific folder of the same format (.txt) categorize them based on names of the files as they would all be under a similar naming convention such as:

day_1_datafile1.txt,
day_1_datafile2.txt,
day_2_datafile1.txt,
day_2_datafile2.txt,

so on and so on.

I would then need to pool the content of each of these files by day into a new file perhaps a json so that I can feed then to html based interface that displays these data visually. The content formatting of these .txt files are subject to change, however the main thing here is to have them organized and pooled in the right way first.

Any advice or perhaps hints anyone could offer is greatly appreciated.

2

There are a couple of file system nodes that might help you with some of this. But doing it in Node-RED you might hit some memory issues as you would probably need to read the files into memory in order to merge them.

In all honesty, this is something probably done more efficiently right now by a shell script which you could easily call from Node-RED. But maybe others will have better ideas.

3

Indeed, in Linux a simple

 cat *.txt > bigfile.text

Would join them in alphabetical order into one large file.