Connection problems: Node red, mosquitto broker and esp32

General
1

I don't know if this is the most suitable forum for this discussion, but it is one of the most helpful ones I know in the business, so maybe there is someone who can help me.

I am developing an Iot project and I usually take many of my beginner's questions to the forum.

Well, my project consists of an ESP32 card, node red and broker mosquitto (both run on a local host on a machine with windows 10). In this project I perform the reading of sensors and the activation of some actuators as relays. I have a well-built and well-functioning node red flow. My problem is in the broker mosquitto: As for the reading of the sensors everything goes well, with the display on the dashboard almost in real time. However, when I am going to activate a relay, it is seconds or minutes until the message leaves the broker and arrives at the board executing the requested. In addition to the latency in the broker-ESP32 sense, I have disconnection problems. It is as if the status messages "ok" did not arrive on time, I believe it is related to the latency that affects the activation of the relays.
What's more, sensor messages range from 5kb to 10kb. While a relay activation message is around 1kb.

I don't understand the reason for the difference in latency.

Why is the reading of the sensors almost momentary and the activation of the relays not?

Again, I don't know if this is the right place, but node red is often used with these technologies, so I believed that someone here could help me.

I'm almost desperate. Any assistance is welcome.

2

Know that this is not normal in any way shape or form.
We run a Mosquitto broker about 3000 miles away and it responds almost instantly.
That said. You clearly have something odd about your setup that is causing the issue.
Two thoughts;

  1. ping -t What does that show over time? (I'm wondering if there is a network issue)
  2. Put a catch node in the flow. What does it show? (We need more information about the core issue).

I have never needed them, but subscribe to $SYS topic to see what you can find. (ie, check the Mosquitto logs).

1 Like
3

So what are your actuator boards ? Are they based on ESP32 units as well ??

When you say it takes a long time to leave the broker - are you actually monitoring the logs (as mentioned above) and do you have a realtime method to monitor when the board actually receive the message in realtime.

You might want to put MQTT-SPY (or equivalent) onto another machine on the network and get it to mimic the actuator to enable you to quantify the delay)

Craig

1 Like
4

+1 to Craig's suggestion, use mqtt-explorer or something similar to monitor the broker to check if you see a delay there. If you see no delay there then the issue must be either the connection between the ESP and the broker or in the ESP itself. Check the mosquito logs to see if there are problems with the connection.

1 Like
5

After you checked the the first leg (Node-RED - Mosquitto) with ESP explorer I think your problem lay on the second leg the ESP itself.

  • what kind of Hardware your relay is implemented?
  • what kind of software are you using? Own source or a custom firmware like Tasmota or ESPEasy?
  • what library are you using?
  • can you monitor/debug your ESP by a serial connection or incircuit debugger.
1 Like
6

@craigcurtin
@Colin
@thebaldgeek
I use a esp32 Node-MCU card. The relays are simple to operate, do not have an individual circuitry, they only recognize the high or low level (3.3V or 0V) of a pin on the Node-MCU board and turn on / off.

I use the Arduino's own IDE adapted for the esp32 board. In the code in question I use the pubsubclient.h library.

And yes, I use serial communication to debug the arrival of message directly from the serial monitor, so the latency is not in the status check, since the serial monitor is a real time.

I'll see if I can share some prints of the broker's behavior with the verbal mode as well as the Esp32 code

7

I checked the broker logs through verbal mode. I really believe it is related to the broker-esp32 connection.

but I have doubts.

I will try to share prints of the behavior

8

Copy/paste the log output here if it looks suspicious.

1 Like
9

see, this is what happens when reading the sensors:

1590161193: Received PUBLISH from ESP32Client (d0, q0, r0, m0, 'event1', ... (5 bytes))
1590161193: Sending PUBLISH to mqtt_2950655f.2e9b7a (d0, q0, r0, m0, 'event1', ... (5 bytes))

The topic event 1 corresponds to the reading of the sensor value. As you can see right after publishing through esp node red already requests a subscriber.

now check this out:

1590161114: Received PUBLISH from mqtt_2950655f.2e9b7a (d0, q1, r0, m5683, 'topic', ... (1 bytes))
1590161114: Sending PUBACK to mqtt_2950655f.2e9b7a (m5683, rc0)
1590161114: Sending PUBLISH to ESP32Client (d0, q0, r0, m0, 'topic', ... (1 bytes))
1590161114: Received PUBLISH from mqtt_2950655f.2e9b7a (d0, q1, r0, m5684, 'topic', ... (1 bytes))
1590161114: Sending PUBACK to mqtt_2950655f.2e9b7a (m5684, rc0)
1590161114: Sending PUBLISH to ESP32Client (d0, q0, r0, m0, 'topic', ... (1 bytes))

it seems that the structure is ok, but when I measured the latency by the serial, it took 40 seconds since the broker left and arrived at the board.

One of the things I realized is the qos. Does it interfere with this speed? could it be one of the problems? the qos of the esp subcriber are in default, 0.
the node red subcriber has qos 2.

finally check this out

1590161210: Received SUBSCRIBE from ESP32Client
1590161210: topic2 (QoS 0)
1590161210: ESP32Client 0 topic2
1590161210: Sending SUBACK to ESP32Client
1590161210: Received PUBLISH from ESP32Client (d0, q0, r0, m0, 'event1', ... (5 bytes))
1590161210: Sending PUBLISH to mqtt_2950655f.2e9b7a (d0, q0, r0, m0, 'event1', ... (5 bytes))
1590161210: Socket error on client ESP32Client, disconnecting.

This disconnect happens out of nowhere, I don't understand if it's related to the keepalive or something.

10

If you are seeing that regularly then that is probably the issue. You should only see the subscribe once when the ESP initially connects, then it should stay connected. If it is continually connecting and disconnecting then that is probably the problem.

11

If you have more than 1 esp connecting with the same clientID you will see this kind of behaviour. Perhaps related?

12

I think not. There are currently only two clients: 1 Esp32 and node-red

13

Is it continually disconnecting? You haven't actually concerned that.
Is it your own Esp code?

14

The pubsub lib is only able to do QoS=0 due to memory and performance considerations. There are branches available doing fake QoS>0 but I do not recommend these. QoS can differ between clients and so that should not be a problem. WD Reboots or client reconnects should not happen (often). WD resets coul be caused by several reasons : power supply, memory leeks... reconnects (wifi or mqtt) can happen if something in your main loop took too much time and do not call client.loop() in time.


void loop()
{
  // put your main code here, to run repeatedly:
  if (!client.connected())
  {
    reconnect();
  }

   client.loop();
}

So first I would check if there are unexpected wifi and/or mqtt disconnects. If you send with qos=2 the message stays on the broker if the connection to your ESP is offline- this is where the delay could happen (because someone must store that message) if it is receiving the broker in time.
Is the wifi signal strength ok (Wrong or bad antenna - the pcb antenna is actually quite good, for external antennas you often have to solder a jumper) and your neighborhood not to crowded?
I do not expect your broker is flooded with messages all to topics your esp is subscribed to (you can monitor this by looking into the $sys topic)
I wold strip down my code to one of the proven example and monitor the messages.

1 Like
15

@Colin
@Christian-Me
this is the code. but I can't identify any errors.

#include <WiFi.h>
#include <PubSubClient.h>

#include <Ultrasonic.h>
 Ultrasonic ultrassom(22,23); 
long distancia;



const char* ssid = "***";
const char* password = "***";
const char* mqtt_server = "***";


static const char* connectionString = "";
long lastMsg = 0; 
char msg[50]; int value = 0;

static bool hasIoTHub = false;


WiFiClient espClient; PubSubClient client(espClient);

void setup()
{
Serial.begin(115200);
String readString;
// Comunicação Serial
pinMode(14, OUTPUT);



setup_wifi();

client.setServer(mqtt_server, 1883); 
client.setCallback(callback); 
}
void callback(char* topic, byte* payload, unsigned int length)
{
Serial.print("Message arrived [");
Serial.print(topic);
Serial.print("] ");
Serial.print((char)payload[0]);

if (strcmp(topic,"topic1")==0){

  if((char)payload[0] == '1')
{

  Serial.print("rele ligado");
  digitalWrite(14, HIGH);
  client.loop();
}
else 
{
  Serial.print("rele desligado");
  digitalWrite(14, LOW);
}

}

}



void setup_wifi()
{

delay(10);
Serial.println();
Serial.print("Conectando a rede ");
Serial.println(ssid); 
WiFi.begin(ssid, password); 
while (WiFi.status() != WL_CONNECTED) 
{
delay(500);
Serial.print(".");
}
Serial.println("");
Serial.println("WiFi conectado");
Serial.println("Endereço de IP: ");
Serial.println(WiFi.localIP()); }
void reconnect() 
{
client.setServer(mqtt_server, 1883); 
while (!client.connected()) 
{
Serial.print("Tentando conectar MQTT...");
if (client.connect("ESP32Client")) 
{
Serial.println("Conectado"); 

}
else
{
Serial.print("falha, rc=");
Serial.print(client.state());
Serial.println(" Tentando novamente em 5 segundos");
delay(5000);
}
}
}


void loop()
{
  
float h = ultrassom.Ranging(CM); 
Serial.print(h);    
char ChuvaString[8]; 
dtostrf(h, 1, 2, ChuvaString); 
Serial.print("Chuva: ");
Serial.println(ChuvaString);
client.publish("event1", ChuvaString); 
client.subscribe("topic");
client.subscribe("topic2");
delay(2000);
 
if (!client.connected())
{ 
reconnect(); 
}
  client.loop(); 

}

An important fact: I carried out tests in the afternoon and realized one thing: the disconnections are at regular intervals, they always occur after a few tens of sensor readings.

It is as if a "delay" is accumulating and in the end it ends in a disconnection

16

I tend to use a fork of the PubSubClient - https://github.com/Imroy/pubsubclient which has been further developed to include new MQTT classes, such as QoS, multiple topics with one (un)subscribe message, etc.
I've had it running on 2 Wemos D1 boards for the past 12 months which speaks & listens to my raspberry running Mosquitto & node-RED, and has been very reliable.

17

At what period?
Also how often are new values being published to "topic" and "topic2"?

18

There are approximately 30 messages between each disconnection. This gives 1 minute since each sensor reading is done every 2 seconds.

"topic" and "topic2" are values ​​dependent on a button on the dashboard. New values ​​are only published if the button is pressed manually.

19

You have got a 2 second pause (delay) in your main loop - if messages are received during that time they will be ignored

Unless you are doing something really fancy i would look in the short term at one of the pre-built distros - ESPhome, ESP Easy, Tasmota etc and see if you see the same problem

It would also be good to be able to see in realtime the output from the Mosquitto broker (in Linux we would just tail the log file) - i have not used it in Windows so someone else will have to comment on that

Craig

1 Like
20

I think this shows the technique you should use for MQTT, note how the loop does not contain a delay, but uses a timer to decide what to do.
Also note that it does not subscribe every time round the loop, but once on connection (and re-connect). I don't know whether that might cause problems as effectively you are adding multiple subscriptions.

1 Like