# Add option to limit number of topics per subscription requests

**URL:** <https://discourse.nodered.org/t/add-option-to-limit-number-of-topics-per-subscription-requests/55879>\
**Category:** Feature Requests\
**Tags:** mqtt\
**Created:** [30 December 2021 09:21 UTC](https://discourse.nodered.org/t/add-option-to-limit-number-of-topics-per-subscription-requests/55879 "2021-12-30T09:21:12Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Barbudor](https://avatars.discourse-cdn.com/v4/letter/b/4af34b/32.png) [@Barbudor](https://discourse.nodered.org/u/Barbudor)\
**Post date:** [30 December 2021 09:21 UTC](https://discourse.nodered.org/t/add-option-to-limit-number-of-topics-per-subscription-requests/55879/1 "2021-12-30T09:21:12Z")

</div>

Hi

As discussed here : [MQTT reconnection failure to Flespi.io - #16 by Barbudor](https://discourse.nodered.org/t/mqtt-reconnection-failure-to-flespi-io/55768/16)  
it appears that some brokers (here it's Flespi (dot) io) have some limitations on the number of topics that can be submitted in a single subscription requests:

> At bottom of page [flespi MQTT broker - MQTT 5, secure, fast, and free](https://flespi.com/mqtt-broker)
> 
> - If using multiple topics per one subscribe/unsubscribe request, broker accepts maximum 16 topics in it

Nodered's MQTT broker node could support an option to specify the maximum number of topic per requests. Subscription requests would then be splitted in multiple requests as needed.

For now, I will work around that limitation by creating multiple MQTT broker nodes, each with a different client ID, and monitoring the number of subscription I attach to each broker node

Thanks for considering the feature.

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [30 December 2021 09:25 UTC](https://discourse.nodered.org/t/add-option-to-limit-number-of-topics-per-subscription-requests/55879/2 "2021-12-30T09:25:52Z")

</div>

Not sure what that means. The Mqtt subscribe message only has one topic in per request. It can include a wildcard symbol which I guess could map to more than x topics but the requester has no way of knowing that. That is information only the broker knows.

---

<div class="post-metadata">

**Author:** ![knolleary](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/knolleary/32/3_2.png) [@knolleary](https://discourse.nodered.org/u/knolleary)\
**Post date:** [30 December 2021 09:34 UTC](https://discourse.nodered.org/t/add-option-to-limit-number-of-topics-per-subscription-requests/55879/3 "2021-12-30T09:34:33Z")

</div>

At a protocol level you can subscribe to multiple topics in a single subscribe packet.

But having checked the code, as far as I can see from my phone screen, we never do that - we always send each node's subscription in its own packet.

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [30 December 2021 09:47 UTC](https://discourse.nodered.org/t/add-option-to-limit-number-of-topics-per-subscription-requests/55879/4 "2021-12-30T09:47:59Z")

</div>

Ah maybe v5 dynamic subscription allows it ? but even so it is well within the control of the user to restrict it.

---

<div class="post-metadata">

**Author:** ![knolleary](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/knolleary/32/3_2.png) [@knolleary](https://discourse.nodered.org/u/knolleary)\
**Post date:** [30 December 2021 09:54 UTC](https://discourse.nodered.org/t/add-option-to-limit-number-of-topics-per-subscription-requests/55879/5 "2021-12-30T09:54:02Z")

</div>

> [@dceejay](#):
>
> maybe v5 dynamic subscription

As far as I could see, even if you pass in an array of topics as part of a dynamic subscription, they are sent as individual requests (although that's the bit I could have overlooked some subtly of on my phone screen)

---

<div class="post-metadata">

**Author:** ![Barbudor](https://avatars.discourse-cdn.com/v4/letter/b/4af34b/32.png) [@Barbudor](https://discourse.nodered.org/u/Barbudor)\
**Post date:** [30 December 2021 10:03 UTC](https://discourse.nodered.org/t/add-option-to-limit-number-of-topics-per-subscription-requests/55879/6 "2021-12-30T10:03:53Z")

</div>

Hi  
Thanks for looking however this is not what the TCP recordings shows  
As you can see on the screenshot below (from Flespi own TCP Proxy recorder), the subscription request includes all my 18 topics.  
I can try to record that as a tcpdump if you like

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/2/6/26d618dc33861e2c1e52246adc2c00c2faabecd1.png)

---

<div class="post-metadata">

**Author:** ![Barbudor](https://avatars.discourse-cdn.com/v4/letter/b/4af34b/32.png) [@Barbudor](https://discourse.nodered.org/u/Barbudor)\
**Post date:** [30 December 2021 10:10 UTC](https://discourse.nodered.org/t/add-option-to-limit-number-of-topics-per-subscription-requests/55879/7 "2021-12-30T10:10:13Z")

</div>

Looking in more details there is indeed a difference between "initial connection" like when I restart the flows or a reconnection.  
Above is the initial connection and as far I can read the binary it is made of independant subscription request per topic (aggregation is just because TCP is a stream)  
However, when a reconnection occurs (which is the problem I have), it looks like the message is different and seems to be aggragated

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/a/b/ab71ea5b7a02821730f9399c2287ac5ef5eb1ce7.png)

---

<div class="post-metadata">

**Author:** ![knolleary](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/knolleary/32/3_2.png) [@knolleary](https://discourse.nodered.org/u/knolleary)\
**Post date:** [30 December 2021 10:18 UTC](https://discourse.nodered.org/t/add-option-to-limit-number-of-topics-per-subscription-requests/55879/8 "2021-12-30T10:18:17Z")

</div>

Ah right. So that is the underlying mqtt client resubscribing after a reconnect.

At first glance, I don't see any [options in that library](https://github.com/mqttjs/MQTT.js#client) to limit how many topics it places in each sub message. And I think this is the first time I've ever come across a broker implementation with such a limit.

This would require driving changes to the upstream library first before we could do anything in Node-RED.

---

<div class="post-metadata">

**Author:** ![bakman2](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bakman2/32/6207_2.png) [@bakman2](https://discourse.nodered.org/u/bakman2)\
**Post date:** [30 December 2021 10:31 UTC](https://discourse.nodered.org/t/add-option-to-limit-number-of-topics-per-subscription-requests/55879/9 "2021-12-30T10:31:17Z")

</div>

> [@Barbudor](#):
>
> For now, I will work around that limitation by creating multiple MQTT broker nodes, each with a different client ID, and monitoring the number of subscription I attach to each broker node

Is this the solution ?

Isn't it a matter of just using the same broker config and use more specificity in the topic you are subscribing to ?  
The limitation is in the request (besides the subscription model limitations).

![Screenshot 2021-12-30 at 11.29.39](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/2/2/22c0de4794caff47221b7e7b889f8e0b20284b4e.jpeg)

---

<div class="post-metadata">

**Author:** ![Barbudor](https://avatars.discourse-cdn.com/v4/letter/b/4af34b/32.png) [@Barbudor](https://discourse.nodered.org/u/Barbudor)\
**Post date:** [30 December 2021 10:37 UTC](https://discourse.nodered.org/t/add-option-to-limit-number-of-topics-per-subscription-requests/55879/10 "2021-12-30T10:37:35Z")

</div>

As an additional test, I unchecked "Clean Session" and disconnected the node-red host for 1 minute.  
This time the reconnection did not send the subscription request (which was expected when restoring the session).  
It would be interresting to check what would happen with a longer break but I prefer to avoid doing this now on this site. I could make a separated test setup within a couple of weeks.

Let me know if you consider the request as closed or would like to hear more from it later.

Thanks a lot

---

<div class="post-metadata">

**Author:** ![Barbudor](https://avatars.discourse-cdn.com/v4/letter/b/4af34b/32.png) [@Barbudor](https://discourse.nodered.org/u/Barbudor)\
**Post date:** [30 December 2021 10:45 UTC](https://discourse.nodered.org/t/add-option-to-limit-number-of-topics-per-subscription-requests/55879/11 "2021-12-30T10:45:27Z")

</div>

Hi @bakman2  
It needs to have less topics subscribed too. so not to make the topics more specific (which would increase the number of needed topic) but on the opposite to topics more generic (putting the # lower in the hierarchy) and use Node-red to parse and redistribute the messages  
That's indeed feasible but would need some more refactoring of my flows than balancing the existing subscriptions across 2 independant configurations to the same broker.

As written above, it seems that disabling Clean Session avoids the re-subscription and may be the alternative. But still I need to test it with longer outages.

---

<div class="post-metadata">

**Author:** ![knolleary](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/knolleary/32/3_2.png) [@knolleary](https://discourse.nodered.org/u/knolleary)\
**Post date:** [30 December 2021 10:47 UTC](https://discourse.nodered.org/t/add-option-to-limit-number-of-topics-per-subscription-requests/55879/12 "2021-12-30T10:47:53Z")

</div>

Clean session will certainly stop the underlying client from resubscribing on connect - but it will mean you don't get any messages queued up for you whilst disconnected.

---

<div class="post-metadata">

**Author:** ![Barbudor](https://avatars.discourse-cdn.com/v4/letter/b/4af34b/32.png) [@Barbudor](https://discourse.nodered.org/u/Barbudor)\
**Post date:** [30 December 2021 11:08 UTC](https://discourse.nodered.org/t/add-option-to-limit-number-of-topics-per-subscription-requests/55879/13 "2021-12-30T11:08:00Z")

</div>

> [@knolleary](#):
>
> Clean session will certainly stop the underlying client from resubscribing on connect

It's the opposite. It's disabling "Clean Session" that stops the re-subscription on re-connect (as the broker has kept the susbcribed topics with the session that is restored). When Clean Session is enabled, it sends that aggregated re-subscription.

And as you pointed, indeed, disabling "Clean Session" is better to get any message queued during the outage.

---

<div class="post-metadata">

**Author:** ![bakman2](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bakman2/32/6207_2.png) [@bakman2](https://discourse.nodered.org/u/bakman2)\
**Post date:** [30 December 2021 11:14 UTC](https://discourse.nodered.org/t/add-option-to-limit-number-of-topics-per-subscription-requests/55879/14 "2021-12-30T11:14:40Z")

</div>

I am this reading the other way around ?

> If using multiple topics **per one** subscribe/unsubscribe **request** , broker accepts maximum 16 topics in it

If you generalize, you will end up with more topics in the request (note that everything after `#` or `+` is a topic)  
If you apply specificity, you will have less topics **per request**

They probably applied this limitation due to the outgoing size per message (256MB = a lot of bandwidth)

---

<div class="post-metadata">

**Author:** ![Barbudor](https://avatars.discourse-cdn.com/v4/letter/b/4af34b/32.png) [@Barbudor](https://discourse.nodered.org/u/Barbudor)\
**Post date:** [30 December 2021 11:34 UTC](https://discourse.nodered.org/t/add-option-to-limit-number-of-topics-per-subscription-requests/55879/15 "2021-12-30T11:34:28Z")

</div>

May be we are not using "generalize" in the same way ?  
As of today I have 18 subscriptions (some have wildcards + or #, some are uniques)  
For example I have 2 MQTT input nodes, 1 for each of the below topics:

```auto
sonnaz/gateway/tele/STATE
sonnaz/teleinfo/tele/STATE

```

I could merge with `sonnaz/+/tele/STATE` and add a switch node to route within Nodered  
As I have 5 similar pair of nodes, such merge would reduce my total number of topics subscribed from 18 to 13, below the limit.  
This is what I call making it more generic while the original one are more specific.  
I could even subscribe to `#` which would be only 1 subscribed topic and do all the parsing and distribution in nodered.

Do we agree ?

The Flespi limit is on the number of topics in the request (where-ever these have wildcards or not), not on the actual number of effective topics addressed by the subscription.

---

<div class="post-metadata">

**Author:** ![bakman2](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bakman2/32/6207_2.png) [@bakman2](https://discourse.nodered.org/u/bakman2)\
**Post date:** [30 December 2021 11:42 UTC](https://discourse.nodered.org/t/add-option-to-limit-number-of-topics-per-subscription-requests/55879/16 "2021-12-30T11:42:17Z")

</div>

Looking at my broker:

![Screenshot 2021-12-30 at 12.39.34](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/2/1/21fec26cb773a71dcabbddd49443340cce594d2c.jpeg)

If I subscribe to `devices/#` = 77 topics ( =1 request)

Or:

![Screenshot 2021-12-30 at 12.39.54](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/b/c/bc22dee3e707cca294585e505d7b577588f78817.jpeg)

If I subscribe to `devices/livingroom/fan/#` = 5 topics ( = 1 request)  
and if you use wildcards, you still get the aggregated number of topics that match.

---

<div class="post-metadata">

**Author:** ![Barbudor](https://avatars.discourse-cdn.com/v4/letter/b/4af34b/32.png) [@Barbudor](https://discourse.nodered.org/u/Barbudor)\
**Post date:** [30 December 2021 12:00 UTC](https://discourse.nodered.org/t/add-option-to-limit-number-of-topics-per-subscription-requests/55879/17 "2021-12-30T12:00:08Z")

</div>

You are not looking at the subscription sent by Nodered (which you could see using tcpdump/wireshark) but the actual number of topics as reported by (I would guess) MQTT Explorer  
Subscribing to `devices/#` includes only 1 topic pattern in the subscription. It is independant from the point that there could be hundreds of topics behind.  
If you have a nodered MQTT input node subscribing to `devices/#`, Nodered has no idea of what are the effective topics behind which are totally dynamic. The request from Nodered would include only 1 topic with the wildcard even if there are (currently) 77 uique topics behind.  
In this system with Flepsi I have 18 MQTT input nodes which implies 18 subscriptions. Reducing this number would mean using more generic topics and routing in Nodered. I have another system with Mosquitto as broker with nearly a hundred of MQTT input nodes, no problems.

I think we are going a bit outside of the original request.

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [30 December 2021 12:35 UTC](https://discourse.nodered.org/t/add-option-to-limit-number-of-topics-per-subscription-requests/55879/18 "2021-12-30T12:35:55Z")

</div>

Indeed - as limiting the number is not part of the MQTT spec - and so far only seems to be part of this broker implementation - it is not obvious if this should be "fixed" by Node-RED or by reading the docs from that broker provider and following their rules.

---

<div class="post-metadata">

**Author:** ![Barbudor](https://avatars.discourse-cdn.com/v4/letter/b/4af34b/32.png) [@Barbudor](https://discourse.nodered.org/u/Barbudor)\
**Post date:** [30 December 2021 13:23 UTC](https://discourse.nodered.org/t/add-option-to-limit-number-of-topics-per-subscription-requests/55879/19 "2021-12-30T13:23:49Z")

</div>

At some point every system may have limitations and having parameters to adjust in orders to allow systems to interoperate is not a bad thing.

But I totally get your point.  
So far I have 2 possible workarounds (disabling Clean Session or balancing my MQTT inputs across 2 configurations) so I won't push for more.

Thanks for your time looking into this.  
And as always for the great work on Nodered 👍

Have a great end of the year and keep safe.

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [30 December 2021 19:05 UTC](https://discourse.nodered.org/t/add-option-to-limit-number-of-topics-per-subscription-requests/55879/20 "2021-12-30T19:05:43Z")

</div>

Hi.

I could have sworn I posted this earlier but I don't see it 🤔

I'll try again...

The mqtt out on mqtt in swords can be sent a message instructing it to disconnect and connect the broker.

If you get this problem again instead of restarting all the flows, perhaps you could try sending the disconnect/connect actions.  
Under the hood I'm unsure if this resubscribes or does a reconnect (been some months since I was in the code) but it's worth a go?

This feature is described in the built in help of the mqtt nodes.

[Next page](https://discourse.nodered.org/t/add-option-to-limit-number-of-topics-per-subscription-requests/55879.md?page=2)