# Node-red-contrib-deduplication

**URL:** <https://discourse.nodered.org/t/node-red-contrib-deduplication/21378>\
**Category:** General\
**Created:** [7 February 2020 05:16 UTC](https://discourse.nodered.org/t/node-red-contrib-deduplication/21378 "2020-02-07T05:16:00Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Iliyas](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/iliyas/32/19045_2.png) [@Iliyas](https://discourse.nodered.org/u/Iliyas)\
**Post date:** [7 February 2020 05:16 UTC](https://discourse.nodered.org/t/node-red-contrib-deduplication/21378/1 "2020-02-07T05:16:00Z")

</div>

Hello,  
I'm working on deduplication node but its not working as it states, if anybody had worked on it please share the flow.  
node-red-contrib-dedupulication-adv

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [7 February 2020 10:30 UTC](https://discourse.nodered.org/t/node-red-contrib-deduplication/21378/2 "2020-02-07T10:30:05Z")

</div>

> [@Iliyas](#):
>
> node-red-contrib-depulication-adv

I presume the spelling there is a mistype, but even spelling as one would expect I can't even find the node. Can you provide the link to it on npmjs, or some other link?

---

<div class="post-metadata">

**Author:** ![Iliyas](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/iliyas/32/19045_2.png) [@Iliyas](https://discourse.nodered.org/u/Iliyas)\
**Post date:** [7 February 2020 10:35 UTC](https://discourse.nodered.org/t/node-red-contrib-deduplication/21378/3 "2020-02-07T10:35:03Z")

</div>

> **[node-red-contrib-deduplicate-adv](https://flows.nodered.org/node/node-red-contrib-deduplicate-adv)**
>
> Node-RED node that filters duplicate messages

  
this is link for the node

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [7 February 2020 10:44 UTC](https://discourse.nodered.org/t/node-red-contrib-deduplication/21378/4 "2020-02-07T10:44:01Z")

</div>

ok, so not node-red-contrib-depulication-adv but node-red-contrib-dedupulicate-adv.

Can you post a simple mini-flow using inject nodes to inject messages showing it failing.

I don't know what you are trying to achieve but have you considered whether the standard RBE node will do what you want?

---

<div class="post-metadata">

**Author:** ![Iliyas](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/iliyas/32/19045_2.png) [@Iliyas](https://discourse.nodered.org/u/Iliyas)\
**Post date:** [7 February 2020 10:50 UTC](https://discourse.nodered.org/t/node-red-contrib-deduplication/21378/5 "2020-02-07T10:50:09Z")

</div>

```auto
[{"id":"78978128.4b3f2","type":"tab","label":"Flow 1","disabled":false,"info":""},{"id":"17d5d1c1.b4b4ae","type":"deduplicate","z":"78978128.4b3f2","name":"","expirypolicy":"keep","registryclass":"","noderole":"standalone","expiry":"5","x":356.57303619384766,"y":312.3818817138672,"wires":[["f1de7d18.dfa2"],["196c735c.9c556d"]]},{"id":"38de3741.26ab08","type":"inject","z":"78978128.4b3f2","name":"","topic":"","payload":"[1,2,2,1]","payloadType":"bin","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":147.57291412353516,"y":304.9583435058594,"wires":[["17d5d1c1.b4b4ae"]]},{"id":"f1de7d18.dfa2","type":"debug","z":"78978128.4b3f2","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":610.5832977294922,"y":271.527774810791,"wires":[]},{"id":"196c735c.9c556d","type":"debug","z":"78978128.4b3f2","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":606.5729522705078,"y":357.5278091430664,"wires":[]}] 

```

this is this flow i had created

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [7 February 2020 11:02 UTC](https://discourse.nodered.org/t/node-red-contrib-deduplication/21378/6 "2020-02-07T11:02:50Z")

</div>

What are you expecting to see and what are you seeing?  
By the way if you give your debug nodes names it is easier to identify the data in the debug window.

---

<div class="post-metadata">

**Author:** ![Iliyas](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/iliyas/32/19045_2.png) [@Iliyas](https://discourse.nodered.org/u/Iliyas)\
**Post date:** [7 February 2020 11:10 UTC](https://discourse.nodered.org/t/node-red-contrib-deduplication/21378/7 "2020-02-07T11:10:34Z")

</div>

I don't want to see duplicate values like I'm injecting array of [1,2,2,1] so the output should be [1,2]

---

<div class="post-metadata">

**Author:** ![knolleary](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/knolleary/32/3_2.png) [@knolleary](https://discourse.nodered.org/u/knolleary)\
**Post date:** [7 February 2020 11:25 UTC](https://discourse.nodered.org/t/node-red-contrib-deduplication/21378/8 "2020-02-07T11:25:33Z")

</div>

The node says it is for deduplicating messages - in other words, stopping a message passing through the node if it has already been seen.

Nowhere does it say it can be used to remove duplicate entries from an array.

I'm not aware of a node that can do what you are asking, so for now you could use a Function node with the code:

```auto
msg.payload = [... new Set(msg.payload)]
return msg;

```

This code puts all of the array values into a new Set object. It then converts the Set back to an array. As Sets cannot contain duplicate values, this automatically removes them for you.

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [7 February 2020 11:58 UTC](https://discourse.nodered.org/t/node-red-contrib-deduplication/21378/9 "2020-02-07T11:58:14Z")

</div>

The message coming in is actually a Buffer not an array, so you will need to modify that slightly if you do actually want a Buffer, not an array.

---

<div class="post-metadata">

**Author:** ![Iliyas](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/iliyas/32/19045_2.png) [@Iliyas](https://discourse.nodered.org/u/Iliyas)\
**Post date:** [7 February 2020 12:24 UTC](https://discourse.nodered.org/t/node-red-contrib-deduplication/21378/10 "2020-02-07T12:24:44Z")

</div>

> [@knolleary](#):
>
> msg.payload = [... new Set(msg.payload)] return msg;

with this thing it is working fine.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [7 April 2020 12:24 UTC](https://discourse.nodered.org/t/node-red-contrib-deduplication/21378/11 "2020-04-07T12:24:45Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.