How to append unique messages

[SiPhyLLC]

I have several MQTT 'devices' sending messages such as "A A B C C D" and "A B D E F F". I would like to append all the unique data over an extended period of time. I know I can use the "split" node to create a series of messages, but I am struggling to figure out how I can append the unique content, resulting in something like "A B C D E F". I have unsuccessfully tried to use a few different palettes, such as deduplicate and combine. Any assistance in a flow will be greatly appreciated.

[zenofmud]

So what if you split the msgs, dump them into an array, sort the array, remove duplicates and build the new msg?

If you do a google search ‘remove du-locates from’ with either ‘array’ or ‘string’ you will find multiple solutions.

[Steve-Mcl]

@SiPhyLLC

I have several MQTT 'devices' sending messages such as "A A B C C D" and "A B D E F F"

Do these devices send single characters or actually like "A A B C C D"?

Can I ask, what send data like this? Is it something you are in control of?

Also, for how long should it "append"?
What happens if you receive A then B then A, would the 2nd A be appended to make ABA or dropped? If appended, then What happens when the string becomes millions of characters?

I think more info is required to fully understand.

Also, what is the overall goal?

[SiPhyLLC]

I actually am using node-red as a dashboard for a computer cluster, reporting a variety of things. Each of the computers runs a cron job that send data to node-red over MQTT. The data in this case is from the Unix command "users", which returns a list of users on the machine. There can be duplicate user names from each, as well between different computers. I want to aggregate all the user names across the entire cluster. So data coming in looks like "steve steve fred beth beth beth", etc. I would like to aggregate the messages over a long period of time (days, weeks, months), but be able to clear it out within the flow.

[SiPhyLLC]

I found a solution using a combination of context, concat and Set().

used split - join to convert the string to an array and fed into a Function with:
let inputUsers = msg.payload;
let storedUsers = global.get("users");
let users = inputUsers.concat(storedUsers);
let uniqueUsers = [...new Set(users)];

msg = { payload: uniqueUsers.toString() };
global.set("users", uniqueUsers)

return msg;

[{"id":"c65eebfa.60e218","type":"split","z":"493cf54c.4c8e8c","name":"","splt":" ","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":500,"y":1040,"wires":[["dbf16bfa.2854d8"]]},{"id":"dbf16bfa.2854d8","type":"join","z":"493cf54c.4c8e8c","name":"","mode":"custom","build":"array","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":650,"y":1040,"wires":[["97bc2c06.bbe79"]]},{"id":"97bc2c06.bbe79","type":"function","z":"493cf54c.4c8e8c","name":"RemoveDups","func":"let inputUsers = msg.payload;\nlet storedUsers = global.get(\"users\");\n\nlet users = inputUsers.concat(storedUsers);\n\nlet uniqueUsers = [...new Set(users)];\n\nmsg = { payload: uniqueUsers.toString() };\n\nglobal.set(\"users\", uniqueUsers)\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":820,"y":1040,"wires":[["7203c340.783ffc"]]},{"id":"7203c340.783ffc","type":"ui_template","z":"493cf54c.4c8e8c","group":"77d09509.b8902c","name":"","order":15,"width":12,"height":2,"format":"<style>\n    #mylog {\n        text-align: center;\n        white-space: normal;\n        word-break: break-word;\n        display: flex;\n        align-items: center;\n        justify-content: center;\n    }\n</style>\n\n<div><h3>All Users (both active and inactive):</h3></div>\n<div id=\"mylog\" ng-bind=\"msg.payload\" contenteditable=\"true\">\n    <!-- payload will go here -->\n</div>","storeOutMessages":true,"fwdInMessages":true,"resendOnRefresh":true,"templateScope":"local","x":1020,"y":1040,"wires":[[]]},{"id":"77d09509.b8902c","type":"ui_group","name":"Status","tab":"693268fc.e0baa8","order":1,"disp":true,"width":"24","collapse":true},{"id":"693268fc.e0baa8","type":"ui_tab","name":"Semi - Charts","icon":"show_chart","order":2,"disabled":false,"hidden":false}]