# Can the JOIN node be with 'reduce sequence' on strings (YES)...and should it?(Maybe...)

**URL:** <https://discourse.nodered.org/t/can-the-join-node-be-with-reduce-sequence-on-strings-yes-and-should-it-maybe/51328>\
**Category:** General\
**Created:** [21 September 2021 10:11 UTC](https://discourse.nodered.org/t/can-the-join-node-be-with-reduce-sequence-on-strings-yes-and-should-it-maybe/51328 "2021-09-21T10:11:29Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![zenofmud](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/zenofmud/32/316_2.png) [@zenofmud](https://discourse.nodered.org/u/zenofmud)\
**Post date:** [21 September 2021 10:11 UTC](https://discourse.nodered.org/t/can-the-join-node-be-with-reduce-sequence-on-strings-yes-and-should-it-maybe/51328/1 "2021-09-21T10:11:29Z")

</div>

Node-RED version: v2.0.5  
Node.js version: v14.16.0  
macOS Big Sur v11.6

I have a file I split on '\n's and then want to remove all lines that start with a `#` and join the remaining lines. Normally I split the files and use a `switch` node to eliminate the ones I don't want and have a `delay` node that sends a 'msg.complete' to the join. For example, if the file is this:

```auto
# test file
line 1
# remove me
line 2
line 3
# remove me too

```

the end result should be

```auto
line 1
line 2
line 3

```

I'm experimenting to see if I can eliminate the `switch` and need for the `delay` to tell the `join` it's all finished. In the example flow below I use an `inject` node with `\\` as the line breaks and split on the `\\`'s but I can't figure out how to ignore the lines that start with `#`.

```auto
[{"id":"3f556a5b754ab050","type":"join","z":"5cba28c9e0847a85","name":"","mode":"reduce","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":true,"timeout":"","count":"","reduceRight":false,"reduceExp":"$contains(payload, '#')\t","reduceInit":"","reduceInitType":"str","reduceFixup":"$string($A+payload)","x":1010,"y":320,"wires":[["70b198c0516212a2"]]},{"id":"fdbc9371a9b02ced","type":"inject","z":"5cba28c9e0847a85","name":"JOIN using 'reduce sequence' and strings example","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"# test file\\\\line 1\\\\# remove me\\\\line 2\\\\line3\\\\# remove me too","payloadType":"str","x":1030,"y":200,"wires":[["89c3b84a1977c75c"]]},{"id":"70b198c0516212a2","type":"debug","z":"5cba28c9e0847a85","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":1010,"y":380,"wires":[]},{"id":"89c3b84a1977c75c","type":"split","z":"5cba28c9e0847a85","name":"","splt":"\\\\","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":1010,"y":260,"wires":[["3f556a5b754ab050"]]}]

```

All the 'expressions' I've used end up with a `code: "T2001"``Error at evaluateNumericExpression (/usr/local/lib/node_modules/node-red/node_modules/jsonata/jsonata.js:4095:25)`

Is it possible to use the 'reduce sequence' with strings and if so are there any examples?

---

<div class="post-metadata">

**Author:** ![UnborN](https://avatars.discourse-cdn.com/v4/letter/u/4491bb/32.png) [@UnborN](https://discourse.nodered.org/u/UnborN)\
**Post date:** [21 September 2021 11:11 UTC](https://discourse.nodered.org/t/can-the-join-node-be-with-reduce-sequence-on-strings-yes-and-should-it-maybe/51328/2 "2021-09-21T11:11:42Z")

</div>

Hi Paul.

I experimented a bit with your flow but didnt find an easy way to do this with switch, join nodes  
I resorted to reading the whole file as a single string and splitting / filtering it later with javascript.

**Test flow**

```auto
[{"id":"1e5b07eab8eded22","type":"file in","z":"4895ea10b4ee9ead","name":"","filename":"C:\\Users\\User\\Desktop\\test.txt","format":"utf8","chunk":false,"sendError":false,"encoding":"none","allProps":true,"x":440,"y":680,"wires":[["99f12f756baecae1","5c2f48f3471eed6e"]]},{"id":"a49f24837328c093","type":"inject","z":"4895ea10b4ee9ead","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":210,"y":680,"wires":[["1e5b07eab8eded22"]]},{"id":"99f12f756baecae1","type":"debug","z":"4895ea10b4ee9ead","name":"1","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":590,"y":620,"wires":[]},{"id":"5c2f48f3471eed6e","type":"function","z":"4895ea10b4ee9ead","name":"","func":"let data = msg.payload.split(\"\\n\")\nlet result = \"\";\n\ndata.forEach( line => {\n if (!line.startsWith('#')) {\n result += line + '\\n';\n }\n});\n\nmsg.payload = result\nreturn msg\n\n","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":690,"y":680,"wires":[["160ac9d61ee64558"]]},{"id":"160ac9d61ee64558","type":"debug","z":"4895ea10b4ee9ead","name":"2","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":850,"y":680,"wires":[]}]

```

---

<div class="post-metadata">

**Author:** ![bakman2](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bakman2/32/6207_2.png) [@bakman2](https://discourse.nodered.org/u/bakman2)\
**Post date:** [21 September 2021 11:31 UTC](https://discourse.nodered.org/t/can-the-join-node-be-with-reduce-sequence-on-strings-yes-and-should-it-maybe/51328/3 "2021-09-21T11:31:25Z")

</div>

You might be able to do it with a single change node:

jsonata  
`$join($split(payload,/[\n]/)[$contains($,/^[^#]/)],"\n")`

In your example the `\n` are not shown (they show up as `\\`) which fails.

```auto
[{"id":"9f65bad80b066e77","type":"debug","z":"c449382138238ad4","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":602,"y":288,"wires":[]},{"id":"c4ec7b907c0ae560","type":"change","z":"c449382138238ad4","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"$join($split(payload,/[\\n]/)[$contains($,/^[^#]/)],\"\\n\")","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":440,"y":288,"wires":[["9f65bad80b066e77"]]},{"id":"f5dcee32f8af1c84","type":"inject","z":"c449382138238ad4","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":108,"y":288,"wires":[["4d2d8aad14ce723f"]]},{"id":"4d2d8aad14ce723f","type":"function","z":"c449382138238ad4","name":"input string","func":"str = `# test file\\n\nline 1\\n\n# remove me\\n\nline 2\\n\nline 3\\n\n# remove me too\\n\n`\nmsg.payload = str\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":262,"y":288,"wires":[["c4ec7b907c0ae560"]]}]

```

---

<div class="post-metadata">

**Author:** ![zenofmud](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/zenofmud/32/316_2.png) [@zenofmud](https://discourse.nodered.org/u/zenofmud)\
**Post date:** [21 September 2021 11:42 UTC](https://discourse.nodered.org/t/can-the-join-node-be-with-reduce-sequence-on-strings-yes-and-should-it-maybe/51328/4 "2021-09-21T11:42:09Z")

</div>

> [@bakman2](#):
>
> In your example the `\n` are not shown (they show up as `\\` ) which fails.

Yes I know the example uses \ because I couldn’t quickly figure out how get the \n into an inject node. But it demonstrates what I want to do. Read a text file, split it into separate lines, ignore some of the lines and join it back to a complete msg.payload.

---

<div class="post-metadata">

**Author:** ![UnborN](https://avatars.discourse-cdn.com/v4/letter/u/4491bb/32.png) [@UnborN](https://discourse.nodered.org/u/UnborN)\
**Post date:** [21 September 2021 11:46 UTC](https://discourse.nodered.org/t/can-the-join-node-be-with-reduce-sequence-on-strings-yes-and-should-it-maybe/51328/5 "2021-09-21T11:46:28Z")

</div>

> [@zenofmud](#):
>
> I couldn’t quickly figure out how get the \n into an inject node

I noticed it also with the inject node.  
i think you can use the Template node in those demo cases and preserve the newlines

```auto
[{"id":"3fec674cc65648a3","type":"inject","z":"4895ea10b4ee9ead","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":330,"y":980,"wires":[["b7d1a91670f78b15"]]},{"id":"b7d1a91670f78b15","type":"template","z":"4895ea10b4ee9ead","name":"","field":"payload","fieldType":"msg","format":"handlebars","syntax":"plain","template":"# test file\nline 1\n# remove me\nline 2\nline 3\n# remove me too","output":"str","x":510,"y":980,"wires":[["a4febe5811ea864f"]]},{"id":"a4febe5811ea864f","type":"debug","z":"4895ea10b4ee9ead","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":690,"y":980,"wires":[]}]

```

---

<div class="post-metadata">

**Author:** ![zenofmud](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/zenofmud/32/316_2.png) [@zenofmud](https://discourse.nodered.org/u/zenofmud)\
**Post date:** [21 September 2021 11:51 UTC](https://discourse.nodered.org/t/can-the-join-node-be-with-reduce-sequence-on-strings-yes-and-should-it-maybe/51328/6 "2021-09-21T11:51:05Z")

</div>

Yeah, but that still doesn’t answer the original question which should have been ‘how to read a text file and remove some lines containing a certain value and have a msg.payload contains the remaking lines?’

This is something I have found a need for several times. Maybe I should think about creating a node to do it….

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [21 September 2021 12:11 UTC](https://discourse.nodered.org/t/can-the-join-node-be-with-reduce-sequence-on-strings-yes-and-should-it-maybe/51328/7 "2021-09-21T12:11:53Z")

</div>

I cant answer the original topic title question (dont do JSONata unless its a simple string thing).

> [@zenofmud](#):
>
> ‘how to read a text file and remove some lines containing a certain value and have a msg.payload contains the remaking lines?’

#### Here are 2 solutions...

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/d/9/d9e73a6e202c70e18560c6d56e88d8c1bb761b82.png)

```auto
[{"id":"89c3b84a1977c75c","type":"split","z":"553814a2.1248ec","name":"","splt":"\\\\","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":1850,"y":1873,"wires":[["a37743a450ae9b2a"]]},{"id":"fdbc9371a9b02ced","type":"inject","z":"553814a2.1248ec","name":"file data","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"# test file\\\\line 1\\\\# remove me\\\\line 2\\\\line3\\\\# remove me too","payloadType":"str","x":1690,"y":1940,"wires":[["89c3b84a1977c75c","83b847e4a32eeeac"]]},{"id":"a37743a450ae9b2a","type":"switch","z":"553814a2.1248ec","name":"","property":"payload","propertyType":"msg","rules":[{"t":"regex","v":"#.*","vt":"str","case":false},{"t":"else"}],"checkall":"true","repair":false,"outputs":2,"x":1970,"y":1873,"wires":[[],["3f556a5b754ab050"]]},{"id":"83b847e4a32eeeac","type":"function","z":"553814a2.1248ec","name":"remove lines starting with #","func":"const lines = msg.payload.split(\"\\\\\\\\\");\nconst result = [];\n\nlines.forEach(function(element) {\n if(!element.startsWith(\"#\")) {\n result.push(element);\n }\n})\n\nmsg.payload = result.join(\"\\n\");\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":1920,"y":1980,"wires":[["2b74f0f7eab6c3d3"]]},{"id":"3f556a5b754ab050","type":"join","z":"553814a2.1248ec","name":"","mode":"custom","build":"array","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"0.2","count":"","reduceRight":false,"reduceExp":"$contains(payload, '#')\t","reduceInit":"","reduceInitType":"str","reduceFixup":"$string($A+payload)","x":1870,"y":1922,"wires":[["e1ca2c2297ce4f6d"]]},{"id":"2b74f0f7eab6c3d3","type":"debug","z":"553814a2.1248ec","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":2210,"y":1980,"wires":[]},{"id":"e1ca2c2297ce4f6d","type":"change","z":"553814a2.1248ec","name":"to string","rules":[{"t":"set","p":"payload","pt":"msg","to":"$join(payload, '\\n')","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":2020,"y":1922,"wires":[["70b198c0516212a2"]]},{"id":"70b198c0516212a2","type":"debug","z":"553814a2.1248ec","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":2210,"y":1922,"wires":[]}]

```

---

<div class="post-metadata">

**Author:** ![bakman2](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bakman2/32/6207_2.png) [@bakman2](https://discourse.nodered.org/u/bakman2)\
**Post date:** [21 September 2021 12:14 UTC](https://discourse.nodered.org/t/can-the-join-node-be-with-reduce-sequence-on-strings-yes-and-should-it-maybe/51328/8 "2021-09-21T12:14:26Z")

</div>

I posted a flow that did provide it ?

---

<div class="post-metadata">

**Author:** ![E1cid](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/e1cid/32/77971_2.png) [@E1cid](https://discourse.nodered.org/u/E1cid)\
**Post date:** [21 September 2021 13:17 UTC](https://discourse.nodered.org/t/can-the-join-node-be-with-reduce-sequence-on-strings-yes-and-should-it-maybe/51328/9 "2021-09-21T13:17:42Z")

</div>

Try this

```auto
[{"id":"fdbc9371a9b02ced","type":"inject","z":"b779de97.b1b46","name":"JOIN using 'reduce sequence' and strings example","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"# test file\\\\line 1\\\\# remove me\\\\line 2\\\\line3\\\\# remove me too","payloadType":"str","x":450,"y":4460,"wires":[["89c3b84a1977c75c"]]},{"id":"89c3b84a1977c75c","type":"split","z":"b779de97.b1b46","name":"","splt":"\\\\","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":430,"y":4520,"wires":[["3f556a5b754ab050"]]},{"id":"3f556a5b754ab050","type":"join","z":"b779de97.b1b46","name":"","mode":"reduce","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":true,"timeout":"","count":"","reduceRight":false,"reduceExp":"$contains(payload, '#') ? $A : $append($A, [payload] ) ","reduceInit":"[]","reduceInitType":"json","reduceFixup":"$join($A, \"\\n\")","x":430,"y":4580,"wires":[["70b198c0516212a2"]]},{"id":"70b198c0516212a2","type":"debug","z":"b779de97.b1b46","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":430,"y":4640,"wires":[]}]

```

to concat strings in JSONata use & not +  
[edit] added the new lines back in.

---

<div class="post-metadata">

**Author:** ![zenofmud](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/zenofmud/32/316_2.png) [@zenofmud](https://discourse.nodered.org/u/zenofmud)\
**Post date:** [21 September 2021 14:00 UTC](https://discourse.nodered.org/t/can-the-join-node-be-with-reduce-sequence-on-strings-yes-and-should-it-maybe/51328/10 "2021-09-21T14:00:22Z")

</div>

@E1cid That is perfect! it works great and removes any delay I had. Time to update a few flows and write this example up! Giving you the credit for the JSONata! (But not till the grandkids are gone)

---

<div class="post-metadata">

**Author:** ![zenofmud](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/zenofmud/32/316_2.png) [@zenofmud](https://discourse.nodered.org/u/zenofmud)\
**Post date:** [22 September 2021 11:15 UTC](https://discourse.nodered.org/t/can-the-join-node-be-with-reduce-sequence-on-strings-yes-and-should-it-maybe/51328/11 "2021-09-22T11:15:57Z")

</div>

Well I had t go back and give @Steve-Mcl the solution. While @E1cid also solves the issue as I was putting together a test and a writeup, I realized that the E1cid solution removes a line that contains the character combination anywhere in the line. While Steve's solution used a 'startsWith' in the test.

BUT that is not the real reason for changing the solution. Speed was the edge.  
For a small file, 200 lines, there was no time difference but when I used a file of 200,000 lines (6MB), the split.join flow took 33 seconds to complete while the function version took `less than a second!`

`Conclusion`  
In this particular situation, while the function node has some overhead, the JSONata has way more when processing large amounts of data (with small amounts of data the diference is negligible.

This was run on a Mac Mini M1 with 16GB of memory using macOS Big Sur.

---

<div class="post-metadata">

**Author:** ![E1cid](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/e1cid/32/77971_2.png) [@E1cid](https://discourse.nodered.org/u/E1cid)\
**Post date:** [22 September 2021 11:32 UTC](https://discourse.nodered.org/t/can-the-join-node-be-with-reduce-sequence-on-strings-yes-and-should-it-maybe/51328/12 "2021-09-22T11:32:40Z")

</div>

Just change the conditional contains to a substring equals.  
To check first character

```auto
[{"id":"fdbc9371a9b02ced","type":"inject","z":"b779de97.b1b46","name":"JOIN using 'reduce sequence' and strings example","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"# test file\\\\li#ne 1\\\\# remove me\\\\line 2\\\\line3\\\\# remove me too","payloadType":"str","x":610,"y":4660,"wires":[["89c3b84a1977c75c"]]},{"id":"89c3b84a1977c75c","type":"split","z":"b779de97.b1b46","name":"","splt":"\\\\","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":590,"y":4720,"wires":[["3f556a5b754ab050"]]},{"id":"3f556a5b754ab050","type":"join","z":"b779de97.b1b46","name":"","mode":"reduce","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":true,"timeout":"","count":"","reduceRight":false,"reduceExp":"$substring(payload,0,1) = \"#\" ? $A : $append($A, [payload] ) ","reduceInit":"[]","reduceInitType":"json","reduceFixup":"$join($A, \"\\n\")","x":590,"y":4780,"wires":[["70b198c0516212a2"]]},{"id":"70b198c0516212a2","type":"debug","z":"b779de97.b1b46","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":590,"y":4840,"wires":[]}]

```

I answered your question "can the join node be used with reduce sequence on strings". You now move the goal post. Oh well never mind.

---

<div class="post-metadata">

**Author:** ![zenofmud](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/zenofmud/32/316_2.png) [@zenofmud](https://discourse.nodered.org/u/zenofmud)\
**Post date:** [22 September 2021 11:45 UTC](https://discourse.nodered.org/t/can-the-join-node-be-with-reduce-sequence-on-strings-yes-and-should-it-maybe/51328/13 "2021-09-22T11:45:05Z")

</div>

@E1cid Oh you did in fact answer the question and, in the process, uncovered a new fact that proved that the split/join can be costlier to use than a function node. You get a gold star! 🌟

---

<div class="post-metadata">

**Author:** ![E1cid](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/e1cid/32/77971_2.png) [@E1cid](https://discourse.nodered.org/u/E1cid)\
**Post date:** [22 September 2021 11:48 UTC](https://discourse.nodered.org/t/can-the-join-node-be-with-reduce-sequence-on-strings-yes-and-should-it-maybe/51328/14 "2021-09-22T11:48:44Z")

</div>

No star required, just change title to what is the fastest way to remove items from an array. Then the correct question is answered.

---

<div class="post-metadata">

**Author:** ![bakman2](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bakman2/32/6207_2.png) [@bakman2](https://discourse.nodered.org/u/bakman2)\
**Post date:** [22 September 2021 14:43 UTC](https://discourse.nodered.org/t/can-the-join-node-be-with-reduce-sequence-on-strings-yes-and-should-it-maybe/51328/15 "2021-09-22T14:43:07Z")

</div>

jsonata is indeed not fast with large arrays, perhaps even quicker is to use an `exec` node with a regular expression, like;

```auto
cat inputfile.txt | grep -e "^[^#]"

```

or if you want to store it somewhere else;

```auto
cat inputfile.txt | grep -e "^[^#]" > output.txt

```

---

<div class="post-metadata">

**Author:** ![zenofmud](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/zenofmud/32/316_2.png) [@zenofmud](https://discourse.nodered.org/u/zenofmud)\
**Post date:** [6 October 2021 14:43 UTC](https://discourse.nodered.org/t/can-the-join-node-be-with-reduce-sequence-on-strings-yes-and-should-it-maybe/51328/16 "2021-10-06T14:43:59Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.