# Read File node returns one line too many?

**URL:** <https://discourse.nodered.org/t/read-file-node-returns-one-line-too-many/68807>\
**Category:** General\
**Created:** [10 October 2022 13:39 UTC](https://discourse.nodered.org/t/read-file-node-returns-one-line-too-many/68807 "2022-10-10T13:39:44Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![jbudd](https://avatars.discourse-cdn.com/v4/letter/j/5f8ce5/32.png) [@jbudd](https://discourse.nodered.org/u/jbudd)\
**Post date:** [10 October 2022 13:39 UTC](https://discourse.nodered.org/t/read-file-node-returns-one-line-too-many/68807/1 "2022-10-10T13:39:44Z")

</div>

I have a file (on Linux) which contains 2 lines of text, 4 words, 20 characters

```auto
$ wc testdata
 2 4 20 testdata
$ cat testdata
A line
Another line

```

Each line is terminated by `\n`, there are no blank lines or spaces at the end:

```auto
$ od -c testdata
0000000 A l i n e \n A n o t h e r l
0000020 i n e \n
0000024

```

The Read FIle node, set to send a message per line, sends 3 messages:

> A line

> Another line

> (empty string)

Is this spurious blank line the expected behaviour?

---

<div class="post-metadata">

**Author:** ![knolleary](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/knolleary/32/3_2.png) [@knolleary](https://discourse.nodered.org/u/knolleary)\
**Post date:** [10 October 2022 13:51 UTC](https://discourse.nodered.org/t/read-file-node-returns-one-line-too-many/68807/2 "2022-10-10T13:51:19Z")

</div>

> [@jbudd](#):
>
> Is this spurious blank line the expected behaviour?

Yes - because you have a newline at the end of the file, so your file ends with a blank line.

---

<div class="post-metadata">

**Author:** ![jbudd](https://avatars.discourse-cdn.com/v4/letter/j/5f8ce5/32.png) [@jbudd](https://discourse.nodered.org/u/jbudd)\
**Post date:** [10 October 2022 13:55 UTC](https://discourse.nodered.org/t/read-file-node-returns-one-line-too-many/68807/3 "2022-10-10T13:55:46Z")

</div>

> [@knolleary](#):
>
> so your file ends with a blank line.

No it doesn't. See the character count and the octal dump. There are two new line characters. There are two lines. The node outputs 3 messages.

Presumably the node identifies the end of line1 by seeing the 1st new line character.  
It identifies the end of line 2 by seeing the 2nd new line character.  
Then it uses the 2nd new line character again to spit out a blank line.

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [10 October 2022 14:02 UTC](https://discourse.nodered.org/t/read-file-node-returns-one-line-too-many/68807/4 "2022-10-10T14:02:21Z")

</div>

I see a first line: `A l i n e` + _plus_ + 2 `\n` newline characters === 3 lines.

---

<div class="post-metadata">

**Author:** ![jbudd](https://avatars.discourse-cdn.com/v4/letter/j/5f8ce5/32.png) [@jbudd](https://discourse.nodered.org/u/jbudd)\
**Post date:** [10 October 2022 14:08 UTC](https://discourse.nodered.org/t/read-file-node-returns-one-line-too-many/68807/5 "2022-10-10T14:08:41Z")

</div>

Sorry, I disagree with that interpretation.  
So does the Linux **wc -l** command which says the file has two lines.

Certainly it is possible, but not generally true that the last line in a file has no `\n`.

I suggest that the node should identify a line by:  
From the first character in the file OR The first character after `\n` or `\n\c`  
To the next `\n` or `\n\c` OR EOF

Edit - If you feed the Read File output to a Write File node, with "add new line to each payload", the output file is not identical to the input. It has `\n\n` at the end.

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [10 October 2022 14:14 UTC](https://discourse.nodered.org/t/read-file-node-returns-one-line-too-many/68807/6 "2022-10-10T14:14:12Z")

</div>

This is a very common thing...

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/6/c/6c7d14941a16a0134114df7c4732583fb00221b9.png)

---

<div class="post-metadata">

**Author:** ![jbudd](https://avatars.discourse-cdn.com/v4/letter/j/5f8ce5/32.png) [@jbudd](https://discourse.nodered.org/u/jbudd)\
**Post date:** [10 October 2022 14:27 UTC](https://discourse.nodered.org/t/read-file-node-returns-one-line-too-many/68807/7 "2022-10-10T14:27:22Z")

</div>

And a selection of languages and command lines which don't count a final `\n` as another line (from a stackexchange answer a few years back)

```auto
$ cat -n testdata | tail -n 1 | cut -f 1
     2
$ awk 'END {print NR}' testdata
2
$ sed 's/.*//' testdata | uniq -c
      2
$ LINECT=0; while read -r LINE; do (( LINECT++ )); done < testdata ; echo $LINECT
2
$ perl -lne '}{ print $.' testdata
2
$ wc -l testdata
2 testdata

```

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [10 October 2022 14:50 UTC](https://discourse.nodered.org/t/read-file-node-returns-one-line-too-many/68807/8 "2022-10-10T14:50:54Z")

</div>

I dont disagree but thats not how this is. Changing this behaviour could feasibly break existing flows.

Something I have always done throughout my years is to add a guard that checks if the line is empty (and dispose of it) so it doesnt really catch me out.

Lets put it this way (devils advocate)

If you read the text file line by line into an array then use `array.join('\n')` it will be faithfully recreated (assuming linux or files systems / OSes that use `\n`). How else would you know there is a newline at the end? That new line might be important to some sub systems or applications.

---

<div class="post-metadata">

**Author:** ![jbudd](https://avatars.discourse-cdn.com/v4/letter/j/5f8ce5/32.png) [@jbudd](https://discourse.nodered.org/u/jbudd)\
**Post date:** [10 October 2022 15:04 UTC](https://discourse.nodered.org/t/read-file-node-returns-one-line-too-many/68807/9 "2022-10-10T15:04:09Z")

</div>

> I dont disagree but thats not how this is.

That's fair enough.

Node-red has nodes to read a file line by line and write a file line by line.  
Surely it should be possible to duplicate a file by reading it with the file read and writing it with the file write, without the need to consider it as an array or a string.

I don't think your example of splitting a string at new line characters is a valid way to count lines in a file.

For me it's a bug, and you can't really fail to fix a bug because people may be relying on it.

Oh, this was once recommended here as a way to obtain the last line of a file. Maybe it used to work differently?

I'll hush up now 🫢

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [10 October 2022 15:13 UTC](https://discourse.nodered.org/t/read-file-node-returns-one-line-too-many/68807/10 "2022-10-10T15:13:17Z")

</div>

> [@jbudd](#):
>
> Surely it should be possible to duplicate a file by reading it with the file read and writing it with the file write, without the need to consider it as an array or a string.

errr - indeed that works ...

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/0/0/00ea6af8b258e74e4e84a2d5aed766b0426b8f4e.png)

```auto
conway : /tmp $ wc a.txt
       2 4 20 a.txt
conway : /tmp $ wc a.new
       2 4 20 a.new

```

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [10 October 2022 15:16 UTC](https://discourse.nodered.org/t/read-file-node-returns-one-line-too-many/68807/11 "2022-10-10T15:16:44Z")

</div>

Indeed - if I have a file `A line` with no \n - then `wc -l` says it has 0 lines... which is perverse...

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [10 October 2022 15:19 UTC](https://discourse.nodered.org/t/read-file-node-returns-one-line-too-many/68807/12 "2022-10-10T15:19:59Z")

</div>

> [@jbudd](#):
>
> I don't think your example of splitting a string at new line characters is a valid way to count lines in a file.

The way I have always look at it is "how many positions can the cursor be at in the file"

In the case of your data - there are 3 positions the cursor can be  
 ![WindowsTerminal_Nvs7f9rCmk](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/0/5/05c2ccb693b5e7c7b92f7cd9e407aa408d6b4bfe.gif)

> [@jbudd](#):
>
> For me it's a bug, and you can't really fail to fix a bug because people may be relying on it

I dont consider it a bug but an implementation detail.

If others feel this is something that should be changed then it would (probably) not be until V4 since it is a breaking change.

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [10 October 2022 15:51 UTC](https://discourse.nodered.org/t/read-file-node-returns-one-line-too-many/68807/13 "2022-10-10T15:51:20Z")

</div>

I agree - my way of thinking is as a byte stream.... but yes ends up same as where the cursor is. You read the \n char that moves the curser to line 3 - then you close the file... - the fact that there is nothing there doesn't mean you are not on line3.

BUT - I can now - if I finally register these words ` read a file line by line and write a file line by line` - see that in that case we do then add an extra `\n` when it's not necessary at the end of the file. I then think it is something that is part of the file out node rather than the in node as (as per my streams thinking - at the point of the final \n in per line mode we don't know that the next character doesn't exist.

So yes I think it could be accepted as an issue.

---

<div class="post-metadata">

**Author:** ![jbudd](https://avatars.discourse-cdn.com/v4/letter/j/5f8ce5/32.png) [@jbudd](https://discourse.nodered.org/u/jbudd)\
**Post date:** [10 October 2022 16:41 UTC](https://discourse.nodered.org/t/read-file-node-returns-one-line-too-many/68807/14 "2022-10-10T16:41:29Z")

</div>

_stops hushing, momentarily_  
I am glad you agree there is an issue, I think you are wrong to ascribe it to the Write File node.

Another programmatic way to read a file a line at a time and count them: C getline()

```auto
$ cat try.c
#include <stdio.h>
#include <stdlib.h>

int main(void)
{
    FILE * fp;
    char * line = NULL;
    size_t len = 0;
    ssize_t read;
    int linesread = 0;

    fp = fopen("/home/pi/testdata", "r");
    if (fp == NULL)
        exit(EXIT_FAILURE);

    while ((read = getline(&line, &len, fp)) != -1) {
        printf("Retrieved line of length %zu:\n", read);
        printf("%s", line);
        linesread++;
    }
    printf("Read %i lines in total\n", linesread);

    fclose(fp);
    if (line)
        free(line);
    exit(EXIT_SUCCESS);
}
$ cc -o try try.c
$ ./try
Retrieved line of length 7:
A line
Retrieved line of length 13:
Another line
Read 2 lines in total

```

And you can remove the final new line from testdata, C still reads 2 lines

```auto
$ printf "%s\n%s" "A line with nl" "Another line no nl" > testdata
$ ./try
Retrieved line of length 15:
A line with nl
Retrieved line of length 18:
Another line no nlRead 2 lines in total

```

This is all Linux based, I don't know what the node[s] do on WIndows, don't have NR installed.

I will try and raise an issue on github

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [10 October 2022 17:17 UTC](https://discourse.nodered.org/t/read-file-node-returns-one-line-too-many/68807/15 "2022-10-10T17:17:22Z")

</div>

Sorry - have to disagree on it being the in node... If you are setting it to return a line at a time - ie split on `\n` then we do read it a char at a time and send the string as soon as we see the `\n` - so we then when we get to the next (non)character we need some way to say "oh actually that was the end of the file" - so need to send null. - It needs to be different to a file with a final line that doesn't end `\n`.

When it gets to the out node we need to reconstruct it - and if the add newline box is checked need to do that for every valid line - so all lines except the last null one, whereas currently we just add a newline regardless.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [9 December 2022 17:17 UTC](https://discourse.nodered.org/t/read-file-node-returns-one-line-too-many/68807/16 "2022-12-09T17:17:47Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.