# Count unique id's from a stream of messages

**URL:** <https://discourse.nodered.org/t/count-unique-ids-from-a-stream-of-messages/46110>\
**Category:** General\
**Created:** [23 May 2021 11:57 UTC](https://discourse.nodered.org/t/count-unique-ids-from-a-stream-of-messages/46110 "2021-05-23T11:57:15Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nice\_to\_have](https://avatars.discourse-cdn.com/v4/letter/n/edb3f5/32.png) [@Nice\_to\_have](https://discourse.nodered.org/u/Nice_to_have)\
**Post date:** [23 May 2021 11:57 UTC](https://discourse.nodered.org/t/count-unique-ids-from-a-stream-of-messages/46110/1 "2021-05-23T11:57:15Z")

</div>

Hi,

I am getting a stream of messages and every messages, contains an ID.  
Some ID's send messages more often then other ID's.  
How do I count the amount of ID's in a certain time, lets say 10 minutes?

---

<div class="post-metadata">

**Author:** ![ghayne](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/ghayne/32/39_2.png) [@ghayne](https://discourse.nodered.org/u/ghayne)\
**Post date:** [23 May 2021 12:11 UTC](https://discourse.nodered.org/t/count-unique-ids-from-a-stream-of-messages/46110/2 "2021-05-23T12:11:31Z")

</div>

You could use a switch node to filter out the ID's and then use this node to count them:

> **[node-red-contrib-counter](https://flows.nodered.org/node/node-red-contrib-counter)**
>
> A Node-RED node to create a counter with messages.

---

<div class="post-metadata">

**Author:** ![Nice\_to\_have](https://avatars.discourse-cdn.com/v4/letter/n/edb3f5/32.png) [@Nice\_to\_have](https://discourse.nodered.org/u/Nice_to_have)\
**Post date:** [23 May 2021 12:34 UTC](https://discourse.nodered.org/t/count-unique-ids-from-a-stream-of-messages/46110/3 "2021-05-23T12:34:09Z")

</div>

But will this also count the following stream:

ID1, ID2, ID3, ID1, ID4, ID4, ID5, ID1  
= 5 unique ID's?

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [23 May 2021 12:34 UTC](https://discourse.nodered.org/t/count-unique-ids-from-a-stream-of-messages/46110/4 "2021-05-23T12:34:59Z")

</div>

> [@Nice\_to\_have](#):
>
> contains an ID.

What does that actually mean? is it the topic? or in the payload? or msg.id?

show me a couple of example payloads (i.e. where is this ID that you want to count)

---

<div class="post-metadata">

**Author:** ![Nice\_to\_have](https://avatars.discourse-cdn.com/v4/letter/n/edb3f5/32.png) [@Nice\_to\_have](https://discourse.nodered.org/u/Nice_to_have)\
**Post date:** [23 May 2021 12:45 UTC](https://discourse.nodered.org/t/count-unique-ids-from-a-stream-of-messages/46110/5 "2021-05-23T12:45:10Z")

</div>

I get the following:

msg.payload.senderMmsi: 244730216  
msg.payload.senderMmsi: 244704000  
msg.payload.senderMmsi: 244355000  
msg.payload.senderMmsi: 538071599  
msg.payload.senderMmsi: 244730216  
msg.payload.senderMmsi: 244355000

Some payload.senderMmsi might be the same, and I do not want to count those twice.  
If I take the above, I want to end up with the value: 4

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [23 May 2021 12:47 UTC](https://discourse.nodered.org/t/count-unique-ids-from-a-stream-of-messages/46110/6 "2021-05-23T12:47:48Z")

</div>

So what you want is count of unique numbers in `msg.payload.senderMmsi`?

---

<div class="post-metadata">

**Author:** ![Nice\_to\_have](https://avatars.discourse-cdn.com/v4/letter/n/edb3f5/32.png) [@Nice\_to\_have](https://discourse.nodered.org/u/Nice_to_have)\
**Post date:** [23 May 2021 12:48 UTC](https://discourse.nodered.org/t/count-unique-ids-from-a-stream-of-messages/46110/7 "2021-05-23T12:48:51Z")

</div>

I'm not the best in explaining my problem 🤣  
but, yes, that i what i am looking for

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [23 May 2021 12:49 UTC](https://discourse.nodered.org/t/count-unique-ids-from-a-stream-of-messages/46110/8 "2021-05-23T12:49:19Z")

</div>

how many of these unique numbers would you expect over the course of time?

---

<div class="post-metadata">

**Author:** ![Nice\_to\_have](https://avatars.discourse-cdn.com/v4/letter/n/edb3f5/32.png) [@Nice\_to\_have](https://discourse.nodered.org/u/Nice_to_have)\
**Post date:** [23 May 2021 12:50 UTC](https://discourse.nodered.org/t/count-unique-ids-from-a-stream-of-messages/46110/9 "2021-05-23T12:50:16Z")

</div>

It can be up to 200

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [23 May 2021 12:55 UTC](https://discourse.nodered.org/t/count-unique-ids-from-a-stream-of-messages/46110/10 "2021-05-23T12:55:26Z")

</div>

Here you go...

![rNHHViKjIb](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/d/b/dbc44e39a4bb73aa293dab9533f9cac8745814fe.gif)

```auto
[{"id":"b127e9dd.ebda88","type":"inject","z":"5e6c8b.7f38b374","name":"1000","props":[{"p":"payload.senderMmsi","v":"1000","vt":"num"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":1200,"y":1088,"wires":[["99bca230.f1e7b"]]},{"id":"aae37861.f01d48","type":"inject","z":"5e6c8b.7f38b374","name":"1001","props":[{"p":"payload.senderMmsi","v":"1001","vt":"num"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":1202,"y":1136,"wires":[["99bca230.f1e7b"]]},{"id":"69f69a4.ba09064","type":"inject","z":"5e6c8b.7f38b374","name":"1002","props":[{"p":"payload.senderMmsi","v":"1002","vt":"num"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":1202,"y":1184,"wires":[["99bca230.f1e7b"]]},{"id":"5e69c1e4.cca27","type":"inject","z":"5e6c8b.7f38b374","name":"1003","props":[{"p":"payload.senderMmsi","v":"1003","vt":"num"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":1202,"y":1232,"wires":[["99bca230.f1e7b"]]},{"id":"99bca230.f1e7b","type":"function","z":"5e6c8b.7f38b374","name":"Unique message counter","func":"var seen = context.get(\"seen\") || {}\nvar count = context.get(\"count\") || 0\nif(!seen[msg.payload.senderMmsi]) {\n count++;\n seen[msg.payload.senderMmsi] = true\n}\n\ncontext.set(\"seen\", seen);\ncontext.set(\"count\", count);\nnode.status({text: \"Inique IDs: \" + count})\nmsg.count = count\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":1452,"y":1088,"wires":[["4a997ca.6faa084","47a5b748.076758"]]},{"id":"4a997ca.6faa084","type":"debug","z":"5e6c8b.7f38b374","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1670,"y":1088,"wires":[]},{"id":"47a5b748.076758","type":"debug","z":"5e6c8b.7f38b374","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"count","targetType":"msg","statusVal":"","statusType":"auto","x":1672,"y":1136,"wires":[]},{"id":"351737b5.60c0a8","type":"comment","z":"5e6c8b.7f38b374","name":"test values","info":"","x":1196,"y":1040,"wires":[]}]

```

---

<div class="post-metadata">

**Author:** ![E1cid](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/e1cid/32/77971_2.png) [@E1cid](https://discourse.nodered.org/u/E1cid)\
**Post date:** [23 May 2021 12:59 UTC](https://discourse.nodered.org/t/count-unique-ids-from-a-stream-of-messages/46110/11 "2021-05-23T12:59:23Z")

</div>

Or create a context storage and log id and time, and then filter out times older than ten minutes and count id's.  
This example gives an individual count on id and a total and a unique  
e.g.  
this is set to a 1min count, the inject creates a random id of 1-4

```auto
[{"id":"a15ba8f.8ca98d8","type":"change","z":"c74669a0.6a34f8","name":"","rules":[{"t":"set","p":"idCount","pt":"flow","to":"$append(\t $flowContext(\"idCount\"),\t [{\"id\":payload.id,\"time\":$millis()}]\t)[time > $millis() - 60*1000]\t","tot":"jsonata"},{"t":"set","p":"count","pt":"msg","to":"$count($flowContext(\"idCount\")[$.id=$$.payload.id])","tot":"jsonata"},{"t":"set","p":"uniqueIdCount","pt":"msg","to":"$count($distinct($flowContext(\"idCount\").[id]))","tot":"jsonata"},{"t":"set","p":"totalcount","pt":"msg","to":"$count($flowContext(\"idCount\"))","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":330,"y":2740,"wires":[["4a9c9eed.fd3bd8"]]},{"id":"f6f7ca7a.e186d8","type":"inject","z":"c74669a0.6a34f8","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{ \"id\": $ceil($random()*4)}","payloadType":"jsonata","x":90,"y":2740,"wires":[["a15ba8f.8ca98d8"]]},{"id":"4a9c9eed.fd3bd8","type":"debug","z":"c74669a0.6a34f8","name":"","active":true,"tosidebar":true,"console":false,"tostatus":true,"complete":"true","targetType":"full","statusVal":"\"id\" & payload.id & \" count = \" & count & \" ALL = \" & totalcount & \" UNI = \" & uniqueIdCount","statusType":"jsonata","x":580,"y":2740,"wires":[]}]

```

---

<div class="post-metadata">

**Author:** ![Nice\_to\_have](https://avatars.discourse-cdn.com/v4/letter/n/edb3f5/32.png) [@Nice\_to\_have](https://discourse.nodered.org/u/Nice_to_have)\
**Post date:** [23 May 2021 13:15 UTC](https://discourse.nodered.org/t/count-unique-ids-from-a-stream-of-messages/46110/12 "2021-05-23T13:15:37Z")

</div>

Thanks Steve, works like a charm.

I use it at in the label next to the home icon.  
The green vessels are vessels who are sailing, and their labels show distance in KM's from my antenna.

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/e/8/e85c19d6529f3f826606661e5cf53b9ed1b966dc.jpeg)

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [23 May 2021 13:44 UTC](https://discourse.nodered.org/t/count-unique-ids-from-a-stream-of-messages/46110/13 "2021-05-23T13:44:32Z")

</div>

Shouldn't vessels be in the blue bit or am I missing something?

---

<div class="post-metadata">

**Author:** ![Nice\_to\_have](https://avatars.discourse-cdn.com/v4/letter/n/edb3f5/32.png) [@Nice\_to\_have](https://discourse.nodered.org/u/Nice_to_have)\
**Post date:** [23 May 2021 13:53 UTC](https://discourse.nodered.org/t/count-unique-ids-from-a-stream-of-messages/46110/14 "2021-05-23T13:53:56Z")

</div>

![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/8/b/8b47d6a019d2d8cc9ce25084e42add4b58b48e46.jpeg)  
😉

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [23 May 2021 14:09 UTC](https://discourse.nodered.org/t/count-unique-ids-from-a-stream-of-messages/46110/15 "2021-05-23T14:09:10Z")

</div>

Ah, they are on blue bits. Long thin blue bits.

---

<div class="post-metadata">

**Author:** ![Nice\_to\_have](https://avatars.discourse-cdn.com/v4/letter/n/edb3f5/32.png) [@Nice\_to\_have](https://discourse.nodered.org/u/Nice_to_have)\
**Post date:** [23 May 2021 16:13 UTC](https://discourse.nodered.org/t/count-unique-ids-from-a-stream-of-messages/46110/16 "2021-05-23T16:13:06Z")

</div>

I have changed the unique message counter a bit as I want to reset the counter.  
I have changed it to:

```auto

var seen = context.get("seen") || {}
var count = global.get("count") || 0

if(!seen[msg.payload.senderMmsi]) {
    count++;
    seen[msg.payload.senderMmsi] = true
}
context.set("seen", seen);
global.set("count", count);
node.status({text: "Inique IDs: " + count})
msg.payload.count = count.toFixed(0)
return msg;

```

I then use a change node to set global.count to 0.  
How ever, when I set global.count to 0, it maybe counts up to 2 or 3 and then freezes.  
Why does it do that?

```auto
[{"id":"99bca230.f1e7b","type":"function","z":"a7b2390e.90d3a8","name":"Unique message counter","func":"\nvar seen = context.get(\"seen\") || {}\nvar count = global.get(\"count\") || 0\n\nif(!seen[msg.payload.senderMmsi]) {\n count++;\n seen[msg.payload.senderMmsi] = true\n}\ncontext.set(\"seen\", seen);\nglobal.set(\"count\", count);\nnode.status({text: \"Inique IDs: \" + count})\nmsg.payload.count = count.toFixed(0)\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":1610,"y":120,"wires":[["2db6983.b048068"]]},{"id":"13307c21.232e74","type":"inject","z":"a7b2390e.90d3a8","name":"Reset Counter","props":[{"p":"payload"}],"repeat":"600","crontab":"","once":true,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":1170,"y":200,"wires":[["2bfe4b27.63e8a4"]]},{"id":"2bfe4b27.63e8a4","type":"change","z":"a7b2390e.90d3a8","name":"Reset Counter","rules":[{"t":"set","p":"count","pt":"global","to":"0","tot":"num"}],"action":"","property":"","from":"","to":"","reg":false,"x":1380,"y":200,"wires":[["89d9519d.fcd6"]]}]

```

---

<div class="post-metadata">

**Author:** ![Nice\_to\_have](https://avatars.discourse-cdn.com/v4/letter/n/edb3f5/32.png) [@Nice\_to\_have](https://discourse.nodered.org/u/Nice_to_have)\
**Post date:** [23 May 2021 16:15 UTC](https://discourse.nodered.org/t/count-unique-ids-from-a-stream-of-messages/46110/17 "2021-05-23T16:15:12Z")

</div>

But when I delete global.count in a change node, it works okay.

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [23 May 2021 16:17 UTC](https://discourse.nodered.org/t/count-unique-ids-from-a-stream-of-messages/46110/18 "2021-05-23T16:17:59Z")

</div>

> [@Nice\_to\_have](#):
>
> How ever, when I set global.count to 0, it maybe counts up to 2 or 3 and then freezes.  
> Why does it do that?

It wont count because `seen` still contains all the `seen` IDs - thus only unseen items are counted

Try this...

```auto
[{"id":"b127e9dd.ebda88","type":"inject","z":"6d01e3e4.6003b4","name":"1000","props":[{"p":"payload.senderMmsi","v":"1000","vt":"num"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":650,"y":1180,"wires":[["99bca230.f1e7b"]]},{"id":"aae37861.f01d48","type":"inject","z":"6d01e3e4.6003b4","name":"1001","props":[{"p":"payload.senderMmsi","v":"1001","vt":"num"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":652,"y":1228,"wires":[["99bca230.f1e7b"]]},{"id":"69f69a4.ba09064","type":"inject","z":"6d01e3e4.6003b4","name":"1002","props":[{"p":"payload.senderMmsi","v":"1002","vt":"num"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":652,"y":1276,"wires":[["99bca230.f1e7b"]]},{"id":"5e69c1e4.cca27","type":"inject","z":"6d01e3e4.6003b4","name":"1003","props":[{"p":"payload.senderMmsi","v":"1003","vt":"num"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":652,"y":1324,"wires":[["99bca230.f1e7b"]]},{"id":"99bca230.f1e7b","type":"function","z":"6d01e3e4.6003b4","name":"Unique message counter","func":"var seen = context.get(\"seen\") || {}\nvar count = context.get(\"count\") || 0\n\nif(msg.topic == \"reset\") {\n seen = {}\n count = 0;\n} else if(!seen[msg.payload.senderMmsi]) {\n count++;\n seen[msg.payload.senderMmsi] = true\n}\n\ncontext.set(\"seen\", seen);\ncontext.set(\"count\", count);\nnode.status({text: \"Inique IDs: \" + count})\nmsg.count = count\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":902,"y":1180,"wires":[["4a997ca.6faa084","47a5b748.076758"]]},{"id":"4a997ca.6faa084","type":"debug","z":"6d01e3e4.6003b4","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1120,"y":1180,"wires":[]},{"id":"47a5b748.076758","type":"debug","z":"6d01e3e4.6003b4","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"count","targetType":"msg","statusVal":"","statusType":"auto","x":1122,"y":1228,"wires":[]},{"id":"351737b5.60c0a8","type":"comment","z":"6d01e3e4.6003b4","name":"test values","info":"","x":640,"y":1080,"wires":[]},{"id":"dc4f6d6d.2ad38","type":"inject","z":"6d01e3e4.6003b4","name":"","props":[{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"reset","x":650,"y":1120,"wires":[["99bca230.f1e7b"]]}]

```

↑ avoids the use of global context

---

<div class="post-metadata">

**Author:** ![Nice\_to\_have](https://avatars.discourse-cdn.com/v4/letter/n/edb3f5/32.png) [@Nice\_to\_have](https://discourse.nodered.org/u/Nice_to_have)\
**Post date:** [23 May 2021 16:23 UTC](https://discourse.nodered.org/t/count-unique-ids-from-a-stream-of-messages/46110/19 "2021-05-23T16:23:46Z")

</div>

This does not reset

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [23 May 2021 16:24 UTC](https://discourse.nodered.org/t/count-unique-ids-from-a-stream-of-messages/46110/20 "2021-05-23T16:24:14Z")

</div>

i edited the code 2 mins ago - try it again  
 ![cvqGJ0sPx6](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/c/9/c95b53d31d049db916a686e3b2256831cd89c86a.gif)

[Next page](https://discourse.nodered.org/t/count-unique-ids-from-a-stream-of-messages/46110.md?page=2)