Reading and Writing into a text file from the function node

General
1

Hi,

I am very new to the node-red family and have been working on a little project recently. I was hoping to find some help here with using the function node.

I was hoping to read two lines at a time from a file stored locally and then compare some values in the lines. depending on the comparisons made, I wanted to write some results from the function node to a text file that would be stored locally as well. Is there a way I can read a line from the function node, and also write the results to a separate file from the same function node?

Is there a possibility that I would be able to accomplish this? and also if you can guide me on how to get it done if it is possible.

Thank you in advance.

2

[{"LN":770901,"F1126":2,"F1185":2,"F1034":1301,"F65":0},{"LN":770901,"F1126":2,"F1185":2,"F1034":1303,"F65":0},{"LN":770901,"F1126":2,"F1185":2,"F1034":1330,"F65":0},{"LN":770901,"F1126":2,"F1185":2,"F1034":1338,"F65":0},{"LN":770032,"F1126":4,"F1185":4,"F1034":1330,"F65":0},{"LN":770032,"F1126":4,"F1185":4,"F1034":1303,"F65":0},{"LN":770032,"F1126":4,"F1185":4,"F1034":1301,"F65":0},{"LN":770009,"F1126":6,"F1185":6,"F1034":1301,"F65":0},{"LN":770009,"F1126":6,"F1185":6,"F1034":1330,"F65":0},{"LN":770004,"F1126":7,"F1185":7,"F1034":1330,"F65":0},{"LN":770004,"F1126":7,"F1185":7,"F1034":1301,"F65":0},{"LN":770004,"F1126":7,"F1185":7,"F1034":1303,"F65":0},{"LN":770008,"F1126":7385,"F1185":8,"F1034":1303,"F65":0},{"LN":770008,"F1126":7385,"F1185":8,"F1034":1301,"F65":0},{"LN":770008,"F1126":7385,"F1185":8,"F1034":1330,"F65":0},{"LN":770006,"F1126":7023,"F1185":12,"F1034":1330,"F65":0},{"LN":770006,"F1126":7023,"F1185":12,"F1034":1301,"F65":0},{"LN":770006,"F1126":7023,"F1185":12,"F1034":1303,"F65":0},{"LN":770002,"F1126":2000,"F1185":770,"F1034":1301,"F65":0},{"LN":770002,"F1126":2000,"F1185":770,"F1034":1338,"F65":0},{"LN":770006,"F1126":2002,"F1185":772,"F1034":1338,"F65":0},{"LN":770006,"F1126":2002,"F1185":772,"F1034":1332,"F65":0},{"LN":770006,"F1126":2002,"F1185":772,"F1034":1333,"F65":0},{"LN":770006,"F1126":2002,"F1185":772,"F1034":1330,"F65":0},{"LN":770006,"F1126":2002,"F1185":772,"F1034":1331,"F65":0},{"LN":770006,"F1126":2002,"F1185":772,"F1034":1301,"F65":0},{"LN":770006,"F1126":2002,"F1185":772,"F1034":1303,"F65":0},{"LN":770006,"F1126":2002,"F1185":772,"F1034":1308,"F65":0}]

This is what the file I am reading would look like.

I want to compare the "LN" value with the next line and if it is the same I write something down the "F1126" and the "F1034" and "F65" values in a certain way into another file say "test". Once the "LN" values change I write a new piece of information in the "test" file and then fill in the "F1126" and the "F1034" and "F65" values again.

I wanted to know if there was a way we could read a two lines, compare values and write to the "test" file accordingly. I am also wondering if it'll be easier if I change the file to CSV values without the field. something like this?

770901,2,2,1301,0
770901,2,2,1303,0
770901,2,2,1330,0
770901,2,2,1338,0
770032,4,4,1330,0
770032,4,4,1303,0
770032,4,4,1301,0
770009,6,6,1301,0
770009,6,6,1330,0
770004,7,7,1330,0
770004,7,7,1301,0
770004,7,7,1303,0
770008,7385,8,1303,0
770008,7385,8,1301,0
770008,7385,8,1330,0
770006,7023,12,1330,0
770006,7023,12,1301,0
770006,7023,12,1303,0
770002,2000,770,1301,0
770002,2000,770,1338,0
770006,2002,772,1338,0
770006,2002,772,1332,0
770006,2002,772,1333,0
770006,2002,772,1330,0
770006,2002,772,1331,0
770006,2002,772,1301,0
770006,2002,772,1303,0
770006,2002,772,1308,0
770006,2002,772,1319,0
770006,2002,772,1321,0
770007,2005,775,1321,0

Please help,
Thank you!

3

To access the filing system from a function node, you need to assign the "fs" library at startup to a global variable. You can do that in the settings.js file. Some instructions can be found in that file.

Then in your function, at the top, simply do

let fs = global.get('fs')
4

But if you are very new to Node-RED that isn’t a very “Node-RED” way of doing it.

A more Node-RED way of doing it would be to direct the bit you want to save to a second output from your node. (see multiple outputs section https://nodered.org/docs/writing-functions#multiple-outputs) and then use a file out node to save it to the file.

1 Like
5

All true indeed. But the question was about reading 2 lines at a time - if the file is very large, the direct route may well be better in this particular case.

6

Another Node-RED way to do it. The file in node can be set to read a file a line at a time - then a batch node can be used to create a batch (array) of two lines - which could then be compared either in a function node (simpler)... or even in a join node set to reduce sequence mode... (but that's above my jsonata pay-grade)

2 Likes
7

Thank you for the reply @TotallyInformation. Like I mentioned, I am new to node Red. Would you be able to guide me on where I can access the settings.js file? and what dies the fs library allow me to do in the function node?

8

thank you for the reply @ukmoose.

So, the file I am working on are results of a query from a database. I will have a set of new results every day and there could be anywhere between 100-200 results. I did try to read one line at a time from the file and failed to do so (I might be doing something wrong). how do you suggest I direct the bit you want to save to a second output from my node?. Will I be able to start from where I left off from the lane?

Thank you.

9

You need to read through the Node-RED docs. The settings.js file is explained there. You will find it in your userDir folder that, by default, will usually be found in ~/.node-red.

The fs library is a Node.js standard library for interacting with the filing system.

10

Thank you for the reply @dceejay I just tried that, and it wouldn't work. I added a delay node just to check if it was spitting out just two lines at a time but it dumps the whole file in a single payload.

11

Thank you @TotallyInformation @dceejay @ukmoose, I will look into it the 'fs' library. I tweaked my query to give me results in a way that I would not have to compare two lines anymore, that will save me time in development. I will just pump new "LN" numbers and get the other fields for that specific 'LN' value. Now, all I would have to do is read one line at a time from file 'A' and write values to a file 'B' from each line. does that make it any simpler? do you have any other idea's how I can approach this?

Thank you again!

12

@KevinPamu - following up on Dave's suggestion

  1. What node are you using to read the file?
  2. Did you look at it's settings to see what yoy can change?
  3. did you feed it into a buffer node like he suggested?
  4. did you add debug nodes so you could verify the outputs of each node?
13

@zenofmud Thanks for the reply and I took Dave's suggestion; I am using the read file node (with one line per message) to read and I used a delay to send one message every 5 seconds which I am storing on another file as a buffer. So, now I have a flow that is able to read one line at a time. I am currently trying to write a function node, that will be able to write to a new file 'B' using data from the line in the buffer.

Thanks for the help.

14

Why not use the file out node? If you have some processing to do on the msg, do it and pass the msg along to the file-out node.

17

You can do what ever you want in the function node. I'm just suggesting that instead of trying to write out to the file in the function node, you build the line you want written out, put it in msg.payload and return that message. Then you have a file-out node connected to the output of the function node.

Then you wont have to worry about the file access in the function node. Make it a little easier.

18

Ah, I see what you mean. I was only considering writing out from the function because I can loop certain things and write to file and stay in the function and finish writing to the file from the same node. Just my thought!

1 Like
19

Hi @zenofmud when I write to a file using the file out node (I'm using the add /n to every line option) I get an extra line at the end which is empty. Any why I can fix this with my file out node?

Thanks!

20

Are you sending just one string to the file out node or are you sending it lots of messages, one for each line? If you are sending it just one string then the remove add /n option, as that is adding the newline on the end. Note that the option does not add /n to every line, it adds /n to each payload sent to it.

21

hi @Colin, Thank you for the reply. I am sending 20-30 lines per day. And towards the end of the day I have to send each of these lines into a query to mine some more data for a day report. however, I get a error message on SQL on the last input because of the extra line.

22

That does not answer the question. When you write to the file out node are you sending multiple messages to that node, or one message with the complete file contents?