# Deduplicate Messages based on specific property

**URL:** <https://discourse.nodered.org/t/deduplicate-messages-based-on-specific-property/77749>\
**Category:** General\
**Created:** [20 April 2023 17:10 UTC](https://discourse.nodered.org/t/deduplicate-messages-based-on-specific-property/77749 "2023-04-20T17:10:17Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![stokes776](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/stokes776/32/66036_2.png) [@stokes776](https://discourse.nodered.org/u/stokes776)\
**Post date:** [20 April 2023 17:10 UTC](https://discourse.nodered.org/t/deduplicate-messages-based-on-specific-property/77749/1 "2023-04-20T17:10:17Z")

</div>

Hello,

I have sensors being heard by several gateways. Those gateways push those messages up to the cloud Node-Red server.

Node-Red then has two very similar messages incoming with the exact same telemetry, but slightly different other properties. I'd like to pick only the 'best' one to proceed forward in the flow.

Example Message 1  
topic: 1234  
{  
sensor\_id: 1234  
temperature: 72  
gateway: r419  
rssi: 40  
}

Example Message 2  
topic: 1234  
{  
sensor\_id: 1234  
temperature: 72  
gateway: r403  
rssi: 67  
}

I'd like only the message with the better RSSI (lower is better) (in this case the first one) to go through.  
I tried the deduplicate node, and am infact using that right now. It deduplicates the messages, but makes no consideration, it seems, for mathmatically choosing which message to pass through, rather it seems time based, in that it just passes whichever one gets there first.

I thought I could use the join node, and maybe I can, but in this same stream I also have hundreds of other sensor transmissions coming through as well. if I join, manual mode, and combine each msg.payload, using the value of the msg.topic, it ends up just choosing the last transmission for a given msg.topic (where msg.topic is the sensor\_id), and not necessarily the best rssi.

Example Message 3  
topic: 1222  
{  
sensor\_id: 1222  
temperature: 65  
gateway: r409  
rssi: 67  
}

If I just join on gateway, unfortunately join just gave me the last transmission as heard by each gateway, which ended up with lost data.

I feel like I am missing something obvious in regards to joining a stream of messages received in an amount of time, based on a key, and then looping over them to choose which one has the best value.

Does anyone else have a similar issue or could give me some advice?

---

<div class="post-metadata">

**Author:** ![stokes776](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/stokes776/32/66036_2.png) [@stokes776](https://discourse.nodered.org/u/stokes776)\
**Post date:** [20 April 2023 17:23 UTC](https://discourse.nodered.org/t/deduplicate-messages-based-on-specific-property/77749/2 "2023-04-20T17:23:29Z")

</div>

I think I found a solution with

> **[node-red-contrib-merge-topic](https://flows.nodered.org/node/node-red-contrib-merge-topic)**
>
> Merges incoming messages into a new message whose payload is an array of the received messages - grouping per topic

---

<div class="post-metadata">

**Author:** ![E1cid](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/e1cid/32/77971_2.png) [@E1cid](https://discourse.nodered.org/u/E1cid)\
**Post date:** [20 April 2023 18:25 UTC](https://discourse.nodered.org/t/deduplicate-messages-based-on-specific-property/77749/3 "2023-04-20T18:25:04Z")

</div>

Heres one way using standard nodes , context, and JSONata

```auto
[{"id":"aa0649dac3dba56c","type":"inject","z":"b9860b4b9de8c8da","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"1234","payload":"{\"sensor_id\":1234,\"temperature\":72,\"gateway\":\"r419\",\"rssi\":40}","payloadType":"json","x":170,"y":360,"wires":[["e3f35f846117a96b"]]},{"id":"e3f35f846117a96b","type":"change","z":"b9860b4b9de8c8da","name":"","rules":[{"t":"set","p":"rssi","pt":"msg","to":"$string($$.payload.rssi)","tot":"jsonata"},{"t":"set","p":"sensors[msg.topic][msg.rssi]","pt":"flow","to":"payload","tot":"msg"},{"t":"set","p":"payload","pt":"msg","to":"sensors[msg.topic]","tot":"flow","dc":true}],"action":"","property":"","from":"","to":"","reg":false,"x":400,"y":480,"wires":[["87ae20b55283ba3b"]]},{"id":"f9ad367d388dfe9d","type":"inject","z":"b9860b4b9de8c8da","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"1222","payload":"{ \"sensor_id\": 1222, \"temperature\": 72, \"gateway\": \"r420\", \"rssi\": 88 }","payloadType":"json","x":170,"y":520,"wires":[["e3f35f846117a96b"]]},{"id":"54028d3c3bc688c7","type":"inject","z":"b9860b4b9de8c8da","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"1234","payload":"{ \"sensor_id\": 1234, \"temperature\": 72, \"gateway\": \"r419\", \"rssi\": 57 }","payloadType":"json","x":170,"y":400,"wires":[["e3f35f846117a96b"]]},{"id":"8ca13bcb4974be68","type":"inject","z":"b9860b4b9de8c8da","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"1222","payload":"{\"sensor_id\":1222,\"temperature\":72,\"gateway\":\"r420\",\"rssi\":44}","payloadType":"json","x":170,"y":560,"wires":[["e3f35f846117a96b"]]},{"id":"87ae20b55283ba3b","type":"trigger","z":"b9860b4b9de8c8da","name":"","op1":"","op2":"","op1type":"nul","op2type":"payl","duration":"2000","extend":false,"overrideDelay":false,"units":"ms","reset":"","bytopic":"all","topic":"topic","outputs":1,"x":420,"y":540,"wires":[["fbced18d73bf89c1"]]},{"id":"fbced18d73bf89c1","type":"change","z":"b9860b4b9de8c8da","name":"","rules":[{"t":"delete","p":"rssi","pt":"msg"},{"t":"delete","p":"sensors[msg.topic]","pt":"flow"},{"t":"set","p":"payload","pt":"msg","to":"$$.payload.*^(rssi)[0]","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":640,"y":460,"wires":[["60cbf1a4e853ad44"]]},{"id":"60cbf1a4e853ad44","type":"debug","z":"b9860b4b9de8c8da","name":"debug 250","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":690,"y":580,"wires":[]}]

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [4 May 2023 18:25 UTC](https://discourse.nodered.org/t/deduplicate-messages-based-on-specific-property/77749/4 "2023-05-04T18:25:42Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.