MQTT open files increasing on "Deploy: Restart Flows"

General
1

There is some background on the original issue in this post, however the question I am asking is changing, so opening a new thread: Mosquitto (MQTT) - Client connection from [ip address] denied access by tcpd - #5 by TotallyInformation

Does anyone know if the number of open files in Mosquitto should increase over time?

After some testing, I have found out how to increase the number reported when running this:

sudo netstat -natp | grep ESTABLISHED.*mosquitto | wc -l

Every time I restart flows or do a full deploy (by this, I mean Deploy: Restart Flows or Deploy: Full Deploy - there is no change if I do any other type of Deploy e.g. modified flows or modified nodes) from my Debian Server (running Mosquitto), the count goes up by 268. Should this happen?
for example, it jumped from 3,582 to 3,850 to 4,118 after subsequent restarts.

Some background detail on the setup is here: MQTT open files keeps increasing - #11 by alex88 - Mosquitto MQTT Broker - Cedalo - Forum

It's like the old connections are not being closed, but new ones are being created on each restart.
Maybe this is normal behaviour...

Summary:
Debian Server running Mosquitto - connections increase by 268 on each "Restart Flows"
Debian NR Env - connections increase by 1 on each restart (despite there being 2x mqtt subscriptions in this Env)
3x Pi Envs - connections do not increase on restart - there are 1x, 2x and 3x mqtt subscriptions (so 6x in total) across the 3x Pis

I can't see a pattern based on Mosquitto Client version numbers or anything else (just now) as to what might be the root cause.

EDIT: additional info
I am running NR 2.2.2
The Debian Server is Debian 11; refer below
image

EDIT2: From early testing the open files count does not seem to increase unless I do one of the 2x deploys, as above. I will keep an eye on it though and report back, if it changes.

2

No, it should not. I think that it indicates a node that is not correctly clearing down mqtt connections. Every node should call a function to clear down any open connections when it gets torn down, either by a deploy or by removal from your flow.

I am not seeing this on my own live installation - I've just tested.

Do you have any non-standard MQTT nodes installed?

3

This is my entire palette on the Debian Server

image
image835Ă—1196 30.3 KB

The "Slave" NR Envs are all setup the same:
image

4

Well, I know that uibuilder v5 doesn't leak anything as I tested that quite extensively.

But I don't recognise all of the nodes you have installed. Do any of them interact with MQTT directly?

5

I don't know which use MQTT TBH, but this wasn't a problem on v 2.2 of NR and I haven't installed any new nodes since upgrading to v2.2.2.

Tomorrow, pending time, I will downgrade from v2.2.2 to v2.2 and retest to see if the problem goes away.

Will report back with the details as soon as I have them.

6

Could you test the latest beta V3.0.0-beta.2

There was a good deal of work done on ensuring MQTT connections are released (also, the mqtt.js lib is updated to latest)

7

Hi @Steve-Mcl

Yes, I can test that first if you prefer.

I have NR installed globally across my 4 devices.

I have only upgraded the Debian server to 2.2.2

Unless you have a better plan, I will start with an upgrade of the Debian server first. Test, and then only upgrade the 3x Pis later, if the problem persists.

Let's set how that goes

1 Like
8

Please feedback your findings.

If it is favourable, we may backport the changes to 2.x branch.

Cheers.

9

Will do. Talk tomorrow.

10

Hey,

I upgraded the Debian 11 server to v3.0.0 beta 2 and the problem has gone away on the Debian Server.

When I run this:

sudo netstat -natp | grep ESTABLISHED.*mosquitto | wc -l

I get 283 open files and on each Full Deploy or Restart Flows, the count stays at 283.

When I restart the 3x Pis, which are still on NR 2.2, the count stays the same, however here is the interesting part. When I restart the other NR Env I have on the Debian Server which should have 2x MQTT subscriptions, the count increase by 1 each time.

Personally I can live it it, as I've set my open file limit to a huge number, however you may have a different view, out of principle.

If you need to me to do any further testing I will.

I'll live a little while with v3.0.0 beta 2 and then downgrade to v2.2.2 if it doesn't work well for me. I'm running a full production envs, so they have to be stable.

11

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.