# Deduplicate payload based on url

**URL:** <https://discourse.nodered.org/t/deduplicate-payload-based-on-url/54296>\
**Category:** General\
**Created:** [25 November 2021 22:50 UTC](https://discourse.nodered.org/t/deduplicate-payload-based-on-url/54296 "2021-11-25T22:50:11Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![steffok](https://avatars.discourse-cdn.com/v4/letter/s/2acd7d/32.png) [@steffok](https://discourse.nodered.org/u/steffok)\
**Post date:** [25 November 2021 22:50 UTC](https://discourse.nodered.org/t/deduplicate-payload-based-on-url/54296/1 "2021-11-25T22:50:11Z")

</div>

I would like deduplicate a payload but my payload has content (which changes) and an url (which stays the same). I would like to deduplicate based on URL

This is the function I'm using for the payload. Is there any chance I can use the node-red-contrib-deduplicate to only check on the specific payload which contains the url? Or is the only possibility to check the entire message?

```auto
var results = "Locatie: " + msg.payload.location + " | Prijstype: " + msg.payload.prijstype + " | Prijs: " + msg.payload.prijs / 100 + "\n \n" + "Beschrijving: " + msg.payload.description + "\n \n" + "Afbeelding: " + msg.payload.PictureUrl + "\n \n" + "AdvertentieURL: " + msg.payload.vipurl ;
msg.payload = results
return msg;

```

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [26 November 2021 07:59 UTC](https://discourse.nodered.org/t/deduplicate-payload-based-on-url/54296/2 "2021-11-26T07:59:26Z")

</div>

Do you mean that you only want to pass messages that have a different URL to the previous message?

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [26 November 2021 09:05 UTC](https://discourse.nodered.org/t/deduplicate-payload-based-on-url/54296/3 "2021-11-26T09:05:24Z")

</div>

If that is what you mean then you can do it with a Filter node (previously called RBE) configured like this

![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/8/f/8fef5dd234b5c5b3e42a49b8a5d07f00edf09c5c.png)

---

<div class="post-metadata">

**Author:** ![steffok](https://avatars.discourse-cdn.com/v4/letter/s/2acd7d/32.png) [@steffok](https://discourse.nodered.org/u/steffok)\
**Post date:** [26 November 2021 12:45 UTC](https://discourse.nodered.org/t/deduplicate-payload-based-on-url/54296/4 "2021-11-26T12:45:53Z")

</div>

The thing is that the URL always changes. So it wil go from A,B,C,D,E,F,G,A,A,H,I,A,J,K... So sometimes a duplicate URL will occur. I want to filter out the double over a long period of time.

Think I need the node-red-contrib-deduplicate but I have no clue how to make it look for a specific part of the payload. I think it only checks for the complete message/payload

edit:

founded a solution. I needed to use "Payload Key Property for analysis" in the node-red-contrib-deduplicate module 🙂 it works now

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [10 December 2021 12:46 UTC](https://discourse.nodered.org/t/deduplicate-payload-based-on-url/54296/5 "2021-12-10T12:46:03Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.