Feature request: read file, set msg.complete to true for last line

Core Development Feature Requests
1

When processing text files via the read file node, where it outputs a msg per line, it would be convenient when msg.complete is set to true for the last line. This makes it easier to group lines using the join node without having to add logic in a function node. This would harmonize the behaviour with that of nodes such as the csv node.

image
image1683Ă—226 21.1 KB

2 Likes
2

That sounds sensible.

3

why not use the join node in auto mode ? It should recognise that the count is set at the end of the file and trigger the join also.

4

Thanks for pointing that out. I ran a quick test, when running the join node in auto mode I cannot set the "Send the message after a number of message parts". My goal is to create batches based on a set number of messages.

I also tried it with the batch node, but this doesn't respond to count nor complete. Example: my text files contains 12 lines. I set the batch node to group by 5 messages. This results in two batches of 5 lines, the last two 2 lines are not processed. I'm looking for three batches (messages) of 5, 5 and 2 lines respectively.

Perhaps I'm looking for two improvements:

  • read file; set complete for the last line, this allows the join node to create the 3 messages.
  • batch node; respond to complete to write out the remaining 2 lines.

Of course, tasks can be accomplished today using the function node in conjunction with the join node. However, it might also be beneficial to consider handling this intuitively within the batch node.

5

Assuming you have one message per line of the file, a join node in manual set to 5 message parts and a timeout seems to work.
I guess you would need a rate limit node to ensure 1 second delay between files.

image
image1138Ă—510 39.8 KB

image
image895Ă—81 10.6 KB

1 Like
6

I can see what you mean. I think it would make sense to enhance the batch node to optionally respond to the count / end of file.

2 Likes
7

Created a Pull Request here - Let batch node terminate "early" if msg.parts set to end of sequence by dceejay · Pull Request #4829 · node-red/node-red · GitHub to add to batch node...

Sadly we have just released 4.0 so may have to wait until 4.1... as it's a new feature.

But of course you can copy the changes locally if you want, in the meantime.

2 Likes