# Duplicate entries from an array

**URL:** <https://discourse.nodered.org/t/duplicate-entries-from-an-array/77891>\
**Category:** General\
**Created:** [24 April 2023 17:03 UTC](https://discourse.nodered.org/t/duplicate-entries-from-an-array/77891 "2023-04-24T17:03:44Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![madmax](https://avatars.discourse-cdn.com/v4/letter/m/8baadc/32.png) [@madmax](https://discourse.nodered.org/u/madmax)\
**Post date:** [24 April 2023 17:03 UTC](https://discourse.nodered.org/t/duplicate-entries-from-an-array/77891/1 "2023-04-24T17:03:44Z")

</div>

Hello!  
Is there a way to delete duplicate entries from an array?  
I hope someone can help me.

---

<div class="post-metadata">

**Author:** ![UnborN](https://avatars.discourse-cdn.com/v4/letter/u/4491bb/32.png) [@UnborN](https://discourse.nodered.org/u/UnborN)\
**Post date:** [24 April 2023 17:09 UTC](https://discourse.nodered.org/t/duplicate-entries-from-an-array/77891/2 "2023-04-24T17:09:04Z")

</div>

hello .. you can use a jsonata expression in a Change node  
using the [$distinct() array function](https://docs.jsonata.org/array-functions#distinct)

---

<div class="post-metadata">

**Author:** ![madmax](https://avatars.discourse-cdn.com/v4/letter/m/8baadc/32.png) [@madmax](https://discourse.nodered.org/u/madmax)\
**Post date:** [24 April 2023 17:18 UTC](https://discourse.nodered.org/t/duplicate-entries-from-an-array/77891/3 "2023-04-24T17:18:02Z")

</div>

thanks for your answer

How do I build this exactly? Do you have an example?

---

<div class="post-metadata">

**Author:** ![UnborN](https://avatars.discourse-cdn.com/v4/letter/u/4491bb/32.png) [@UnborN](https://discourse.nodered.org/u/UnborN)\
**Post date:** [24 April 2023 17:27 UTC](https://discourse.nodered.org/t/duplicate-entries-from-an-array/77891/4 "2023-04-24T17:27:55Z")

</div>

Here you go

```auto
[{"id":"dd264254b20f70e8","type":"inject","z":"54efb553244c241f","name":"fake data","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"Heroes","payload":"[{\"name\":\"Molecule Man\",\"age\":29,\"secretIdentity\":\"Dan Jukes\",\"powers\":[\"Radiation resistance\",\"Turning tiny\",\"Radiation blast\"]},{\"name\":\"Molecule Man\",\"age\":29,\"secretIdentity\":\"Dan Jukes\",\"powers\":[\"Radiation resistance\",\"Turning tiny\",\"Radiation blast\"]},{\"name\":\"Molecule Man\",\"age\":29,\"secretIdentity\":\"Dan Jukes\",\"powers\":[\"Radiation resistance\",\"Turning tiny\",\"Radiation blast\"]},{\"name\":\"Molecule Man\",\"age\":29,\"secretIdentity\":\"Dan Jukes\",\"powers\":[\"Radiation resistance\",\"Turning tiny\",\"Radiation blast\"]},{\"name\":\"Madame Uppercut\",\"age\":39,\"secretIdentity\":\"Jane Wilson\",\"powers\":[\"Million tonne punch\",\"Damage resistance\",\"Superhuman reflexes\"]},{\"name\":\"Eternal Flame\",\"age\":1000000,\"secretIdentity\":\"Unknown\",\"powers\":[\"Immortality\",\"Heat Immunity\",\"Inferno\",\"Teleportation\",\"Interdimensional travel\"]}]","payloadType":"json","x":280,"y":2220,"wires":[["0b423c4865359f2f","b8de59a4d2bb94d8"]]},{"id":"962cc58d57493e06","type":"debug","z":"54efb553244c241f","name":"debug 2","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":720,"y":2220,"wires":[]},{"id":"0b423c4865359f2f","type":"change","z":"54efb553244c241f","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"$distinct(payload)","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":520,"y":2220,"wires":[["962cc58d57493e06"]]},{"id":"b8de59a4d2bb94d8","type":"debug","z":"54efb553244c241f","name":"debug 1","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":430,"y":2160,"wires":[]}]

```

Debug 1 shows payload is array of 6 objects (where some are duplicates)  
Debug 2 shows the result (3 unique objects) after the `$distinct()`

---

<div class="post-metadata">

**Author:** ![madmax](https://avatars.discourse-cdn.com/v4/letter/m/8baadc/32.png) [@madmax](https://discourse.nodered.org/u/madmax)\
**Post date:** [24 April 2023 18:23 UTC](https://discourse.nodered.org/t/duplicate-entries-from-an-array/77891/5 "2023-04-24T18:23:35Z")

</div>

Hello!

Thanks for that.

and is it also possible to compare only one value?

for example  
msg.payload.name

---

<div class="post-metadata">

**Author:** ![UnborN](https://avatars.discourse-cdn.com/v4/letter/u/4491bb/32.png) [@UnborN](https://discourse.nodered.org/u/UnborN)\
**Post date:** [24 April 2023 20:42 UTC](https://discourse.nodered.org/t/duplicate-entries-from-an-array/77891/6 "2023-04-24T20:42:03Z")

</div>

$distinct compares the array elements .. even if they are deep nested objects for differences  
and returns the unique ones.

in your requirement you need to check only the `name` and if the name is repeated many times but for example the age is different .. return what ? the first match ? some other condition ?

do you have some real sample data and how the output should be ?

---

<div class="post-metadata">

**Author:** ![madmax](https://avatars.discourse-cdn.com/v4/letter/m/8baadc/32.png) [@madmax](https://discourse.nodered.org/u/madmax)\
**Post date:** [25 April 2023 04:52 UTC](https://discourse.nodered.org/t/duplicate-entries-from-an-array/77891/7 "2023-04-25T04:52:37Z")

</div>

I have an array with two values each.

and it should only be a comparison and if this occurs once in a while, it should then be deleted so that only one remains.

As in your example, only the name should be compared and should it occur again, the double should be completely awake.

---

<div class="post-metadata">

**Author:** ![cameo69](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/cameo69/32/66593_2.png) [@cameo69](https://discourse.nodered.org/u/cameo69)\
**Post date:** [25 April 2023 11:43 UTC](https://discourse.nodered.org/t/duplicate-entries-from-an-array/77891/8 "2023-04-25T11:43:56Z")

</div>

Sounds like this could help

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/0/1/010f774d1657e82902e4e2e9d1fa0639880c10a5.png)

```auto
[{"id":"eb621e3ae7914a09","type":"inject","z":"6313bd35b2395547","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"[[1,2],[\"a\",\"fwfe\"],[\"a\",\"2nd value\"]]","payloadType":"jsonata","x":230,"y":520,"wires":[["ad1d215a26a024ba"]]},{"id":"30b8ac740d4af8dc","type":"inject","z":"6313bd35b2395547","name":"reset","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":230,"y":460,"wires":[["8cce1c633269dad8"]]},{"id":"06d062de94092412","type":"debug","z":"6313bd35b2395547","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":1210,"y":520,"wires":[]},{"id":"ad1d215a26a024ba","type":"split","z":"6313bd35b2395547","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":410,"y":520,"wires":[["374ce67e472c8a12"]]},{"id":"518494695464316f","type":"inject","z":"6313bd35b2395547","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"[[\"name\",\"val\"],[\"a\",\"3rd value\"]]","payloadType":"jsonata","x":230,"y":580,"wires":[["ad1d215a26a024ba"]]},{"id":"8cce1c633269dad8","type":"change","z":"6313bd35b2395547","name":"","rules":[{"t":"delete","p":"list","pt":"flow"}],"action":"","property":"","from":"","to":"","reg":false,"x":440,"y":460,"wires":[[]]},{"id":"374ce67e472c8a12","type":"function","z":"6313bd35b2395547","name":"","func":"if (typeof msg.payload == \"object\" && Array.isArray(msg.payload) && msg.payload.length == 2) {\n let key = msg.payload[0];\n let value = msg.payload[1];\n let list = flow.get(\"list\") ?? {};\n if (!(key in list)) {\n list[key] = value;\n flow.set(\"list\", list);\n }\n}\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":600,"y":520,"wires":[["b1720a07c4c39ac8"]]},{"id":"b1720a07c4c39ac8","type":"change","z":"6313bd35b2395547","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"list","tot":"flow"}],"action":"","property":"","from":"","to":"","reg":false,"x":790,"y":520,"wires":[["5f55e78ce1953230","06d062de94092412"]]},{"id":"5f55e78ce1953230","type":"function","z":"6313bd35b2395547","name":"convert to array","func":"msg.payload = Object.entries(msg.payload);\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":1020,"y":580,"wires":[["06d062de94092412"]]}]

```

It takes before the split array of arrays, e.g. `[["name","value"], ["a","fwfe"], ["a","2nd value"] ]`.  
It collects all those tuples in `flow.list`, if some name comes up again then it is discarded.  
Press reset to delete flow.list to start fresh.

Node `convert to array` is optional, just in case you need an array at the end again.

EDIT:  
Just in case... some say map is faster than objects, the same in flow.map1 with a js Map:

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/d/e/dec76a09c6fb54db19408b8c4885ccc2aa73b928.png)

```auto
[{"id":"94ca4e2013ba9dcc","type":"inject","z":"6313bd35b2395547","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"[[1,2],[\"a\",\"fwfe\"],[\"a\",\"2nd value\"]]","payloadType":"jsonata","x":230,"y":760,"wires":[["6f8c34f6d9cc8c01"]]},{"id":"a1026b531409faba","type":"inject","z":"6313bd35b2395547","name":"reset","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":230,"y":700,"wires":[["31530d0f2073d6b5"]]},{"id":"0c73b9a08bda54cd","type":"debug","z":"6313bd35b2395547","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":1210,"y":760,"wires":[]},{"id":"6f8c34f6d9cc8c01","type":"split","z":"6313bd35b2395547","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":410,"y":760,"wires":[["2308e3a27ef36599"]]},{"id":"9c468332867198cb","type":"inject","z":"6313bd35b2395547","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"[[\"name\",\"val\"],[\"a\",\"3rd value\"]]","payloadType":"jsonata","x":230,"y":820,"wires":[["6f8c34f6d9cc8c01"]]},{"id":"31530d0f2073d6b5","type":"change","z":"6313bd35b2395547","name":"","rules":[{"t":"delete","p":"map1","pt":"flow"}],"action":"","property":"","from":"","to":"","reg":false,"x":450,"y":700,"wires":[[]]},{"id":"2308e3a27ef36599","type":"function","z":"6313bd35b2395547","name":"","func":"if (typeof msg.payload == \"object\" && Array.isArray(msg.payload) && msg.payload.length == 2) {\n let key = msg.payload[0];\n let value = msg.payload[1];\n let map1 = flow.get(\"map1\") ?? new Map();\n if (!(map1.has(key))) {\n map1.set(key, value);\n flow.set(\"map1\", map1);\n }\n}\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":600,"y":760,"wires":[["7343c92659e635e4"]]},{"id":"7343c92659e635e4","type":"change","z":"6313bd35b2395547","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"map1","tot":"flow"}],"action":"","property":"","from":"","to":"","reg":false,"x":790,"y":760,"wires":[["0c73b9a08bda54cd","ddb37998fa45b4f7"]]},{"id":"ddb37998fa45b4f7","type":"function","z":"6313bd35b2395547","name":"convert to array","func":"msg.payload = Array.from(msg.payload, ([name, value]) => ([name, value]));\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":1020,"y":820,"wires":[["0c73b9a08bda54cd"]]}]

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [24 June 2023 11:44 UTC](https://discourse.nodered.org/t/duplicate-entries-from-an-array/77891/9 "2023-06-24T11:44:29Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.