Can the JOIN node be with 'reduce sequence' on strings (YES)...and should it?(Maybe...)

[zenofmud]

Node-RED version: v2.0.5
Node.js version: v14.16.0
macOS Big Sur v11.6

I have a file I split on '\n's and then want to remove all lines that start with a # and join the remaining lines. Normally I split the files and use a switch node to eliminate the ones I don't want and have a delay node that sends a 'msg.complete' to the join. For example, if the file is this:

# test file
line 1
# remove me
line 2
line 3
# remove me too

the end result should be

line 1
line 2
line 3

I'm experimenting to see if I can eliminate the switch and need for the delay to tell the join it's all finished. In the example flow below I use an inject node with \\ as the line breaks and split on the \\'s but I can't figure out how to ignore the lines that start with #.

[{"id":"3f556a5b754ab050","type":"join","z":"5cba28c9e0847a85","name":"","mode":"reduce","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":true,"timeout":"","count":"","reduceRight":false,"reduceExp":"$contains(payload, '#')\t","reduceInit":"","reduceInitType":"str","reduceFixup":"$string($A+payload)","x":1010,"y":320,"wires":[["70b198c0516212a2"]]},{"id":"fdbc9371a9b02ced","type":"inject","z":"5cba28c9e0847a85","name":"JOIN using 'reduce sequence' and strings example","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"# test file\\\\line 1\\\\# remove me\\\\line 2\\\\line3\\\\# remove me too","payloadType":"str","x":1030,"y":200,"wires":[["89c3b84a1977c75c"]]},{"id":"70b198c0516212a2","type":"debug","z":"5cba28c9e0847a85","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":1010,"y":380,"wires":[]},{"id":"89c3b84a1977c75c","type":"split","z":"5cba28c9e0847a85","name":"","splt":"\\\\","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":1010,"y":260,"wires":[["3f556a5b754ab050"]]}]

All the 'expressions' I've used end up with a code: "T2001" Error at evaluateNumericExpression (/usr/local/lib/node_modules/node-red/node_modules/jsonata/jsonata.js:4095:25)

Is it possible to use the 'reduce sequence' with strings and if so are there any examples?

[UnborN]

Hi Paul.

I experimented a bit with your flow but didnt find an easy way to do this with switch, join nodes
I resorted to reading the whole file as a single string and splitting / filtering it later with javascript.

Test flow

[{"id":"1e5b07eab8eded22","type":"file in","z":"4895ea10b4ee9ead","name":"","filename":"C:\\Users\\User\\Desktop\\test.txt","format":"utf8","chunk":false,"sendError":false,"encoding":"none","allProps":true,"x":440,"y":680,"wires":[["99f12f756baecae1","5c2f48f3471eed6e"]]},{"id":"a49f24837328c093","type":"inject","z":"4895ea10b4ee9ead","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":210,"y":680,"wires":[["1e5b07eab8eded22"]]},{"id":"99f12f756baecae1","type":"debug","z":"4895ea10b4ee9ead","name":"1","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":590,"y":620,"wires":[]},{"id":"5c2f48f3471eed6e","type":"function","z":"4895ea10b4ee9ead","name":"","func":"let data = msg.payload.split(\"\\n\")\nlet result = \"\";\n\ndata.forEach( line => {\n    if (!line.startsWith('#')) {\n        result += line + '\\n';\n    }\n});\n\nmsg.payload = result\nreturn msg\n\n","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":690,"y":680,"wires":[["160ac9d61ee64558"]]},{"id":"160ac9d61ee64558","type":"debug","z":"4895ea10b4ee9ead","name":"2","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":850,"y":680,"wires":[]}]

[bakman2]

You might be able to do it with a single change node:

jsonata
$join($split(payload,/[\n]/)[$contains($,/^[^#]/)],"\n")

In your example the \n are not shown (they show up as \\) which fails.

[{"id":"9f65bad80b066e77","type":"debug","z":"c449382138238ad4","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":602,"y":288,"wires":[]},{"id":"c4ec7b907c0ae560","type":"change","z":"c449382138238ad4","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"$join($split(payload,/[\\n]/)[$contains($,/^[^#]/)],\"\\n\")","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":440,"y":288,"wires":[["9f65bad80b066e77"]]},{"id":"f5dcee32f8af1c84","type":"inject","z":"c449382138238ad4","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":108,"y":288,"wires":[["4d2d8aad14ce723f"]]},{"id":"4d2d8aad14ce723f","type":"function","z":"c449382138238ad4","name":"input string","func":"str = `# test file\\n\nline 1\\n\n# remove me\\n\nline 2\\n\nline 3\\n\n# remove me too\\n\n`\nmsg.payload = str\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":262,"y":288,"wires":[["c4ec7b907c0ae560"]]}]

[zenofmud]

@bakman2

In your example the \n are not shown (they show up as \\ ) which fails.

Yes I know the example uses \ because I couldn’t quickly figure out how get the \n into an inject node. But it demonstrates what I want to do. Read a text file, split it into separate lines, ignore some of the lines and join it back to a complete msg.payload.

[UnborN]

@zenofmud

I couldn’t quickly figure out how get the \n into an inject node

I noticed it also with the inject node.
i think you can use the Template node in those demo cases and preserve the newlines

[{"id":"3fec674cc65648a3","type":"inject","z":"4895ea10b4ee9ead","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":330,"y":980,"wires":[["b7d1a91670f78b15"]]},{"id":"b7d1a91670f78b15","type":"template","z":"4895ea10b4ee9ead","name":"","field":"payload","fieldType":"msg","format":"handlebars","syntax":"plain","template":"# test file\nline 1\n# remove me\nline 2\nline 3\n# remove me too","output":"str","x":510,"y":980,"wires":[["a4febe5811ea864f"]]},{"id":"a4febe5811ea864f","type":"debug","z":"4895ea10b4ee9ead","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":690,"y":980,"wires":[]}]

[zenofmud]

Yeah, but that still doesn’t answer the original question which should have been ‘how to read a text file and remove some lines containing a certain value and have a msg.payload contains the remaking lines?’

This is something I have found a need for several times. Maybe I should think about creating a node to do it….

[Steve-Mcl]

I cant answer the original topic title question (dont do JSONata unless its a simple string thing).

@zenofmud

‘how to read a text file and remove some lines containing a certain value and have a msg.payload contains the remaking lines?’

Here are 2 solutions...

[{"id":"89c3b84a1977c75c","type":"split","z":"553814a2.1248ec","name":"","splt":"\\\\","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":1850,"y":1873,"wires":[["a37743a450ae9b2a"]]},{"id":"fdbc9371a9b02ced","type":"inject","z":"553814a2.1248ec","name":"file data","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"# test file\\\\line 1\\\\# remove me\\\\line 2\\\\line3\\\\# remove me too","payloadType":"str","x":1690,"y":1940,"wires":[["89c3b84a1977c75c","83b847e4a32eeeac"]]},{"id":"a37743a450ae9b2a","type":"switch","z":"553814a2.1248ec","name":"","property":"payload","propertyType":"msg","rules":[{"t":"regex","v":"#.*","vt":"str","case":false},{"t":"else"}],"checkall":"true","repair":false,"outputs":2,"x":1970,"y":1873,"wires":[[],["3f556a5b754ab050"]]},{"id":"83b847e4a32eeeac","type":"function","z":"553814a2.1248ec","name":"remove lines starting with #","func":"const lines = msg.payload.split(\"\\\\\\\\\");\nconst result = [];\n\nlines.forEach(function(element) {\n    if(!element.startsWith(\"#\")) {\n        result.push(element);\n    }\n})\n\nmsg.payload = result.join(\"\\n\");\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":1920,"y":1980,"wires":[["2b74f0f7eab6c3d3"]]},{"id":"3f556a5b754ab050","type":"join","z":"553814a2.1248ec","name":"","mode":"custom","build":"array","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"0.2","count":"","reduceRight":false,"reduceExp":"$contains(payload, '#')\t","reduceInit":"","reduceInitType":"str","reduceFixup":"$string($A+payload)","x":1870,"y":1922,"wires":[["e1ca2c2297ce4f6d"]]},{"id":"2b74f0f7eab6c3d3","type":"debug","z":"553814a2.1248ec","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":2210,"y":1980,"wires":[]},{"id":"e1ca2c2297ce4f6d","type":"change","z":"553814a2.1248ec","name":"to string","rules":[{"t":"set","p":"payload","pt":"msg","to":"$join(payload, '\\n')","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":2020,"y":1922,"wires":[["70b198c0516212a2"]]},{"id":"70b198c0516212a2","type":"debug","z":"553814a2.1248ec","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":2210,"y":1922,"wires":[]}]

[bakman2]

I posted a flow that did provide it ?

[E1cid]

Try this

[{"id":"fdbc9371a9b02ced","type":"inject","z":"b779de97.b1b46","name":"JOIN using 'reduce sequence' and strings example","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"# test file\\\\line 1\\\\# remove me\\\\line 2\\\\line3\\\\# remove me too","payloadType":"str","x":450,"y":4460,"wires":[["89c3b84a1977c75c"]]},{"id":"89c3b84a1977c75c","type":"split","z":"b779de97.b1b46","name":"","splt":"\\\\","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":430,"y":4520,"wires":[["3f556a5b754ab050"]]},{"id":"3f556a5b754ab050","type":"join","z":"b779de97.b1b46","name":"","mode":"reduce","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":true,"timeout":"","count":"","reduceRight":false,"reduceExp":"$contains(payload, '#') ? $A : $append($A, [payload] ) ","reduceInit":"[]","reduceInitType":"json","reduceFixup":"$join($A, \"\\n\")","x":430,"y":4580,"wires":[["70b198c0516212a2"]]},{"id":"70b198c0516212a2","type":"debug","z":"b779de97.b1b46","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":430,"y":4640,"wires":[]}]

to concat strings in JSONata use & not +
[edit] added the new lines back in.

[zenofmud]

@E1cid That is perfect! it works great and removes any delay I had. Time to update a few flows and write this example up! Giving you the credit for the JSONata! (But not till the grandkids are gone)

[zenofmud]

Well I had t go back and give @Steve-Mcl the solution. While @E1cid also solves the issue as I was putting together a test and a writeup, I realized that the E1cid solution removes a line that contains the character combination anywhere in the line. While Steve's solution used a 'startsWith' in the test.

BUT that is not the real reason for changing the solution. Speed was the edge.
For a small file, 200 lines, there was no time difference but when I used a file of 200,000 lines (6MB), the split.join flow took 33 seconds to complete while the function version took less than a second!

Conclusion
In this particular situation, while the function node has some overhead, the JSONata has way more when processing large amounts of data (with small amounts of data the diference is negligible.

This was run on a Mac Mini M1 with 16GB of memory using macOS Big Sur.

[E1cid]

Just change the conditional contains to a substring equals.
To check first character

[{"id":"fdbc9371a9b02ced","type":"inject","z":"b779de97.b1b46","name":"JOIN using 'reduce sequence' and strings example","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"# test file\\\\li#ne 1\\\\# remove me\\\\line 2\\\\line3\\\\# remove me too","payloadType":"str","x":610,"y":4660,"wires":[["89c3b84a1977c75c"]]},{"id":"89c3b84a1977c75c","type":"split","z":"b779de97.b1b46","name":"","splt":"\\\\","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":590,"y":4720,"wires":[["3f556a5b754ab050"]]},{"id":"3f556a5b754ab050","type":"join","z":"b779de97.b1b46","name":"","mode":"reduce","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":true,"timeout":"","count":"","reduceRight":false,"reduceExp":"$substring(payload,0,1) = \"#\" ? $A : $append($A, [payload] ) ","reduceInit":"[]","reduceInitType":"json","reduceFixup":"$join($A, \"\\n\")","x":590,"y":4780,"wires":[["70b198c0516212a2"]]},{"id":"70b198c0516212a2","type":"debug","z":"b779de97.b1b46","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":590,"y":4840,"wires":[]}]

I answered your question "can the join node be used with reduce sequence on strings". You now move the goal post. Oh well never mind.

[zenofmud]

@E1cid Oh you did in fact answer the question and, in the process, uncovered a new fact that proved that the split/join can be costlier to use than a function node. You get a gold star!

[E1cid]

No star required, just change title to what is the fastest way to remove items from an array. Then the correct question is answered.

[bakman2]

jsonata is indeed not fast with large arrays, perhaps even quicker is to use an exec node with a regular expression, like;

cat inputfile.txt | grep -e "^[^#]"

or if you want to store it somewhere else;

cat inputfile.txt | grep -e "^[^#]" > output.txt