# How to append unique messages

**URL:** <https://discourse.nodered.org/t/how-to-append-unique-messages/45146>\
**Category:** General\
**Created:** [3 May 2021 20:59 UTC](https://discourse.nodered.org/t/how-to-append-unique-messages/45146 "2021-05-03T20:59:53Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![SiPhyLLC](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/siphyllc/32/38906_2.png) [@SiPhyLLC](https://discourse.nodered.org/u/SiPhyLLC)\
**Post date:** [3 May 2021 20:59 UTC](https://discourse.nodered.org/t/how-to-append-unique-messages/45146/1 "2021-05-03T20:59:53Z")

</div>

I have several MQTT 'devices' sending messages such as "A A B C C D" and "A B D E F F". I would like to append all the unique data over an extended period of time. I know I can use the "split" node to create a series of messages, but I am struggling to figure out how I can append the unique content, resulting in something like "A B C D E F". I have unsuccessfully tried to use a few different palettes, such as deduplicate and combine. Any assistance in a flow will be greatly appreciated.

---

<div class="post-metadata">

**Author:** ![zenofmud](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/zenofmud/32/316_2.png) [@zenofmud](https://discourse.nodered.org/u/zenofmud)\
**Post date:** [3 May 2021 22:09 UTC](https://discourse.nodered.org/t/how-to-append-unique-messages/45146/2 "2021-05-03T22:09:49Z")

</div>

So what if you split the msgs, dump them into an array, sort the array, remove duplicates and build the new msg?

If you do a google search ‘remove du-locates from’ with either ‘array’ or ‘string’ you will find multiple solutions.

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [3 May 2021 22:38 UTC](https://discourse.nodered.org/t/how-to-append-unique-messages/45146/3 "2021-05-03T22:38:08Z")

</div>

> [@SiPhyLLC](#):
>
> I have several MQTT 'devices' sending messages such as "A A B C C D" and "A B D E F F"

Do these devices send single characters or actually like "A A B C C D"?

Can I ask, what send data like this? Is it something you are in control of?

Also, for how long should it "append"?  
What happens if you receive A then B then A, would the 2nd A be appended to make ABA or dropped? If appended, then What happens when the string becomes millions of characters?

I think more info is required to fully understand.

Also, what is the overall goal?

---

<div class="post-metadata">

**Author:** ![SiPhyLLC](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/siphyllc/32/38906_2.png) [@SiPhyLLC](https://discourse.nodered.org/u/SiPhyLLC)\
**Post date:** [4 May 2021 13:34 UTC](https://discourse.nodered.org/t/how-to-append-unique-messages/45146/4 "2021-05-04T13:34:38Z")

</div>

I actually am using node-red as a dashboard for a computer cluster, reporting a variety of things. Each of the computers runs a cron job that send data to node-red over MQTT. The data in this case is from the Unix command "users", which returns a list of users on the machine. There can be duplicate user names from each, as well between different computers. I want to aggregate all the user names across the entire cluster. So data coming in looks like "steve steve fred beth beth beth", etc. I would like to aggregate the messages over a long period of time (days, weeks, months), but be able to clear it out within the flow.

 ![Hardware_Monitoring_Dashboard](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/3/1/31d7b8fda5c534e12f076682aa20581eb65c6eb3.png)  
 ![Node-RED](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/b/a/baa50ec7617a644b54c747c2fae9836696dc05e4.png)

---

<div class="post-metadata">

**Author:** ![SiPhyLLC](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/siphyllc/32/38906_2.png) [@SiPhyLLC](https://discourse.nodered.org/u/SiPhyLLC)\
**Post date:** [4 May 2021 15:20 UTC](https://discourse.nodered.org/t/how-to-append-unique-messages/45146/5 "2021-05-04T15:20:03Z")

</div>

I found a solution using a combination of context, concat and Set().

used split - join to convert the string to an array and fed into a Function with:  
let inputUsers = msg.payload;  
let storedUsers = global.get("users");  
let users = inputUsers.concat(storedUsers);  
let uniqueUsers = [...new Set(users)];

msg = { payload: uniqueUsers.toString() };  
global.set("users", uniqueUsers)

return msg;

```auto
[{"id":"c65eebfa.60e218","type":"split","z":"493cf54c.4c8e8c","name":"","splt":" ","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":500,"y":1040,"wires":[["dbf16bfa.2854d8"]]},{"id":"dbf16bfa.2854d8","type":"join","z":"493cf54c.4c8e8c","name":"","mode":"custom","build":"array","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":650,"y":1040,"wires":[["97bc2c06.bbe79"]]},{"id":"97bc2c06.bbe79","type":"function","z":"493cf54c.4c8e8c","name":"RemoveDups","func":"let inputUsers = msg.payload;\nlet storedUsers = global.get(\"users\");\n\nlet users = inputUsers.concat(storedUsers);\n\nlet uniqueUsers = [...new Set(users)];\n\nmsg = { payload: uniqueUsers.toString() };\n\nglobal.set(\"users\", uniqueUsers)\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":820,"y":1040,"wires":[["7203c340.783ffc"]]},{"id":"7203c340.783ffc","type":"ui_template","z":"493cf54c.4c8e8c","group":"77d09509.b8902c","name":"","order":15,"width":12,"height":2,"format":"<style>\n #mylog {\n text-align: center;\n white-space: normal;\n word-break: break-word;\n display: flex;\n align-items: center;\n justify-content: center;\n }\n</style>\n\n<div><h3>All Users (both active and inactive):</h3></div>\n<div id=\"mylog\" ng-bind=\"msg.payload\" contenteditable=\"true\">\n <!-- payload will go here -->\n</div>","storeOutMessages":true,"fwdInMessages":true,"resendOnRefresh":true,"templateScope":"local","x":1020,"y":1040,"wires":[[]]},{"id":"77d09509.b8902c","type":"ui_group","name":"Status","tab":"693268fc.e0baa8","order":1,"disp":true,"width":"24","collapse":true},{"id":"693268fc.e0baa8","type":"ui_tab","name":"Semi - Charts","icon":"show_chart","order":2,"disabled":false,"hidden":false}]

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [18 May 2021 15:20 UTC](https://discourse.nodered.org/t/how-to-append-unique-messages/45146/6 "2021-05-18T15:20:20Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.