Need help rebuilding home automation raspberry pi

I spent several hours building a machine with Bookworm installed and did not gain a single step forward. Everything was done with the guide and the responses all seem correct. I certainly agree with you regarding the simple test. What does it really prove? Is there some magic signal between the mqttout/in? And when I click on timestamp button, a timestamp appears magically in the debug pane! I would guess that the flow is correct but the part of the flow that I think is missing is the radio signal from the router to the device and the state msg from the device back to the router and back to the RPi . And I have to discount that possibility because the old system running Buster is sitting right here using the same radio system and working fine. I can't run them at the same time because I don't want to have to use different i/p addresses.
In retrospect, I should never have asked my original question!

That seems to confirm that your broker is accepting the connection from Node-red and you can publish and subscribe.

It does not confirm that it accepts the connection from the "device".
Perhaps your next step is to look into the logging offered by mosquitto to see if connections are being refused.

Hmm.

for "jbudd", I agree with your closing statement above and will search out the logging features of mosquitto .

In an effort to solve this issue, here is the latest status. Remember the first version of this system was created over two years ago from the RandomNerdTutorials course, “Smart home_RaspberryPi. Etc” v1.3. I am wanting to build another system for another application and it was suggested here on this forum that I would be better off going with the latest versions of the software and hardware involved. The course was also upgraded by the authors to v1.7 which is the guide I used to build this version. The original version is still working using a RPi3B+, updated/upgraded regularly. The new version is using a RPi4B with the latest image from RPi Imager. The software is identified in several of the screenshots.
The flow (simple.json) has been reduced to simply demonstrate the problem. The problem seems to have something to do in the flow that prevents the message from the switch from getting to the mqtt output node. All the nodes in this system with the exception of the simple “test1” were exported/imported from the original system. Updates were made to reflect the newer versions. The simple “test1” will produce an output into the “debug” pane of the node red window and it the reason for me saying that is working. The devices controlled in the system are all Sonoff basic switches programmed manually with the correct topic and names.
The problem is that none of the switches respond to the new system but still wok fine with the old system. I would like to think that one single item is at fault here. The simple four node test works but I am not sure how much of the connection between the mqtt nodes is actually used. The RPi4 is connected to the router via ethernet and it is using the same network as the original system.
I have all of the available logs running on Mosquitto but I am not sure what they are really telling me. If someone could enlighten me on the use of the logs, I might just find the error right he in front of me. The screenshots that I have included were taken from the terminal using the command “mosquito -v”. I can see when I make an input to the “test1” flow but I never see any activity when I input to the other flow.
screenshots with description
simple.json (6.9 KB)

Thanks for looking

[Admin edit] Fixed formatting to allow attachment to be accessed

OK one of a couple of problems - on each of the sonoff devices you have to tell them to communicate with a MQTT node - you do this by supplying the IP address of the node - if you have a 2nd box now configured it will have a different IP address than the original.

So you will have to login to each sonoff device and update where it talks to MQTT - i assume you have loaded an alternate firmware onto the Sonoffs as i do not recall any MQTT support in the default firmware (but have not used it or the app for years)

Presumably if they are talking purely MQTT and not Home Assistant protocol (as you do not mention that at all in the above) they point to the old IP address on the old machine.

You can create an MQTT in Node on the new box and configure it to point to the old IP address where the MQTT broker was and see if you see any messages coming in.

If not then you will need to login to the old box at the terminal/SSH level (more information needed as to how NR is running on that box and whether it does indeed run Home Assistant (and if so which version and install method) and you can then monitor the live data in the MQTT server

Craig

Your screenshot says it is not available in my region. You can just paste it inline. Also please paste the flow inline here. Make sure you only export those nodes that are necessary to see the problem. In order to make code readable and usable it is necessary to surround your code with three backticks (also known as a left quote or backquote ```)

``` 
   code goes here 
```

See this post for more details - How to share code or flow json

Could you expand on the above statement?
Programmed manually.
Do you mean they have been flashed with Tasmota or some other firmware?

So in summary... you have a set of SonOff switches "talking" to a RPi-4 via MQTT
and the RPi-4 is running Node-RED and Mosquitto (as your MQTT broker)?

Here is the flow file from your attachment

[
    {
        "id": "689a8ea47644053d",
        "type": "tab",
        "label": "Flow 1",
        "disabled": false,
        "info": "",
        "env": []
    },
    {
        "id": "7318524b661c6311",
        "type": "inject",
        "z": "689a8ea47644053d",
        "name": "ON",
        "props": [
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "on",
        "x": 90,
        "y": 80,
        "wires": [
            [
                "8d4d8023772c2e4f"
            ]
        ]
    },
    {
        "id": "b93e8f4590ef8f7b",
        "type": "inject",
        "z": "689a8ea47644053d",
        "name": "OFF",
        "props": [
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "off",
        "x": 90,
        "y": 140,
        "wires": [
            [
                "8d4d8023772c2e4f"
            ]
        ]
    },
    {
        "id": "8d4d8023772c2e4f",
        "type": "ui_switch",
        "z": "689a8ea47644053d",
        "name": "Hot Water switch",
        "label": "Hot Water switch",
        "tooltip": "",
        "group": "59214996.9e7fa8",
        "order": 12,
        "width": 0,
        "height": 0,
        "passthru": true,
        "decouple": "false",
        "topic": "topic",
        "topicType": "msg",
        "style": "",
        "onvalue": "On",
        "onvalueType": "str",
        "onicon": "",
        "oncolor": "",
        "offvalue": "Off",
        "offvalueType": "str",
        "officon": "",
        "offcolor": "",
        "animate": false,
        "className": "",
        "x": 290,
        "y": 100,
        "wires": [
            [
                "759c50bb87126704"
            ]
        ]
    },
    {
        "id": "4cf8810714871578",
        "type": "mqtt in",
        "z": "689a8ea47644053d",
        "name": "home/office/4004_05/state",
        "topic": "home/office/4004_05/state",
        "qos": "2",
        "datatype": "auto-detect",
        "broker": "db9e6b85.5d9138",
        "nl": false,
        "rap": true,
        "rh": 0,
        "inputs": 0,
        "x": 530,
        "y": 180,
        "wires": [
            [
                "49ad72d76484bccf",
                "8aa44fc853573bc0"
            ]
        ]
    },
    {
        "id": "759c50bb87126704",
        "type": "mqtt out",
        "z": "689a8ea47644053d",
        "name": "home/office/4004_05",
        "topic": "home/office/4004_05",
        "qos": "",
        "retain": "",
        "respTopic": "",
        "contentType": "",
        "userProps": "",
        "correl": "",
        "expiry": "",
        "broker": "db9e6b85.5d9138",
        "x": 300,
        "y": 180,
        "wires": []
    },
    {
        "id": "49ad72d76484bccf",
        "type": "ui_text",
        "z": "689a8ea47644053d",
        "group": "59214996.9e7fa8",
        "order": 13,
        "width": 0,
        "height": 0,
        "name": "Hot Water XX",
        "label": "Hot Water XX",
        "format": "{{msg.payload}}",
        "layout": "row-left",
        "className": "",
        "style": false,
        "font": "",
        "fontSize": 16,
        "color": "#000000",
        "x": 840,
        "y": 180,
        "wires": []
    },
    {
        "id": "8aa44fc853573bc0",
        "type": "debug",
        "z": "689a8ea47644053d",
        "name": "Hot water Zz",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "payload",
        "targetType": "msg",
        "statusVal": "",
        "statusType": "auto",
        "x": 830,
        "y": 220,
        "wires": []
    },
    {
        "id": "d65d0e3cda1f65b5",
        "type": "inject",
        "z": "689a8ea47644053d",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "",
        "payloadType": "date",
        "x": 220,
        "y": 360,
        "wires": [
            [
                "0af0f777b78897b1"
            ]
        ]
    },
    {
        "id": "3db64d5c4994f476",
        "type": "debug",
        "z": "689a8ea47644053d",
        "name": "debug XX",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "payload",
        "targetType": "msg",
        "statusVal": "",
        "statusType": "auto",
        "x": 720,
        "y": 360,
        "wires": []
    },
    {
        "id": "3c80922e8ee33541",
        "type": "mqtt in",
        "z": "689a8ea47644053d",
        "name": "",
        "topic": "test1",
        "qos": "2",
        "datatype": "auto-detect",
        "broker": "db9e6b85.5d9138",
        "nl": false,
        "rap": true,
        "rh": 0,
        "inputs": 0,
        "x": 550,
        "y": 360,
        "wires": [
            [
                "3db64d5c4994f476"
            ]
        ]
    },
    {
        "id": "0af0f777b78897b1",
        "type": "mqtt out",
        "z": "689a8ea47644053d",
        "name": "",
        "topic": "test1",
        "qos": "",
        "retain": "",
        "respTopic": "",
        "contentType": "",
        "userProps": "",
        "correl": "",
        "expiry": "",
        "broker": "db9e6b85.5d9138",
        "x": 390,
        "y": 360,
        "wires": []
    },
    {
        "id": "59214996.9e7fa8",
        "type": "ui_group",
        "name": "4004",
        "tab": "4c1cfcd6.cb35c4",
        "order": 1,
        "disp": true,
        "width": "6",
        "collapse": false,
        "className": ""
    },
    {
        "id": "db9e6b85.5d9138",
        "type": "mqtt-broker",
        "name": "localhost:1883",
        "broker": "localhost",
        "port": "1883",
        "clientid": "",
        "autoConnect": true,
        "usetls": false,
        "protocolVersion": "5",
        "keepalive": "60",
        "cleansession": true,
        "autoUnsubscribe": true,
        "birthTopic": "",
        "birthQos": "0",
        "birthPayload": "",
        "birthMsg": {},
        "closeTopic": "",
        "closeQos": "0",
        "closePayload": "",
        "closeMsg": {},
        "willTopic": "",
        "willQos": "0",
        "willPayload": "",
        "willMsg": {},
        "userProps": "",
        "sessionExpiry": ""
    },
    {
        "id": "4c1cfcd6.cb35c4",
        "type": "ui_tab",
        "name": "HOME",
        "icon": "dashboard",
        "disabled": false,
        "hidden": false
    },
    {
        "id": "e18fd8c4be1a533e",
        "type": "global-config",
        "env": [],
        "modules": {
            "node-red-dashboard": "3.6.6"
        }
    }
]

If you add an mqtt In node subscribing to home/office/4004_05 you can check that the message is getting published ok.

Can you tell by looking at something on the end device is receiving the command?
Can you tell whether the end device has connected to your broker?
If you don't know then look in the mosquitto log and it will tell you. Run something like
sudo tail -f -n 100 /var/log/mosquitto/mosquitto.log

Here is a snippet from my mosquitto.log, after restarting the Pi

1774009259: mosquitto version 2.0.11 starting
1774009256: Config loaded from /etc/mosquitto/mosquitto.conf.
1774009256: Opening ipv4 listen socket on port 1883.
1774009256: Opening ipv6 listen socket on port 1883.
1774009256: mosquitto version 2.0.11 running

1774009267: New connection from 127.0.0.1:59978 on port 1883.
1774009268: New client connected from 127.0.0.1:59978 as nodered5d728e8efb19a941 (p2, c1, k60, u'admin').

1774009269: New connection from 192.168.1.100:65239 on port 1883.
1774009269: New client connected from 192.168.1.100:65239 as ESP32C3-AC:XX:XX:XX:XX:XX (p2, c1, k15, u'admin').

First lines show Mosquitto is running and accepting connections on port 1883.

Timestamps are Unix timestamps: seconds since the creation of the world.
unixtimestamp.com tells me these are roughly Fri Mar 20 2026 12:20:59 GMT+0000, ie now-ish.

Then we see a connection from the broker computer itself - the IP is 127.0.0.1 or localhost.
Since the username begins with "nodered" we can tell the connection is from Node-red, and it is using a username and password - username 'admin'.

Next a connection from an ESP32 microcontroller at 192.168.1.100, also using username 'admin'.

I changed the mqtt broker settings on another Node-red machine (wrong username/password).
Now I see multiple failed connection attempts in the logfile

1774009920: New connection from 192.168.1.27:42514 on port 1883.
1774009920: Client <unknown> disconnected, not authorised.

Thanks to all for looking. First of all, to answer a few common queries. The Sonoff devices were flashed with alternate firmware that also came from pages RNTutorials. I have not used anything from Home Assistant /Home Automation/Tasmoto or any other platform. This firmware does define the i/p address of the broker which is the same device running NodeRed , Mosquitto, i.e., theRPi.
I see that the flow was added by some kind admin but I thought I had done the same thing. Anyway the screenshots have been added here;


This one shows what happened when I manually sent the time stamp in the simple four node flow that also produces an output in the debug pane. I also manually triggered the ON button in the "home/office/4004/" node with no effect in that data stream. This is the primary problem as I see it. Why doesn't that msg get passed from the mqtt out node.
The data stream shown was obtained on the terminal after entering the command "mosquitto -v"
as soon as the system started running. It runs continuously and should show any inputs.
The one issue that I can not seem to resolve is this very issue. At the output of the mqtt node, where does the message go. In my mind it would to the router via the ethernet cable and then via wifi to the sonoff switch. If the switch receives the signal and functions, it would send the state message back to the router, ethernet cable and to the input mqtt and to the broker. In this set up , it DOES NOT do that . I have the switch right here on the bench and it is silent. That is also why I think the simple four node example doesn't really prove anything in this case.
And just to reiterate, the subject switch works fine with the original system.

This show just shows data similar to "jbudd" the started few seconds but does seem to show so
some communication with the switch and the test1 nodes.

The last shot just shows the gui but notice the orange outline on the timestamp button, it does produce a response in the debug pane.
My next effort is being directed to learn the use of "MQTT EXPLORER" .
Regards
Charles

I dont know about everyone else - but i am thoroughly confused with what you are showing us - is this from the new system you are building or the old system - or a combination of the two.

I think you have a basic misunderstanding of MQTT

MQTT (broadly) operates around the principal of a central controller (the broker) - this broker enables devices to attach to it to either send or receive messages (or both) - known as subscribing

A device listens to the MQTT broker for messages that interest it on specific topics
A device with information to share sends that information on a specific topic or channel

These pieces of information are not randomly broadcast around the network - they are relayed to specific devices that are connected to the channel that the infromation is meant for.

You can have two identical MQTT brokers on two identical machines - with identical topics/channels on each broker - however a device that has been configure to talk to Broker A does not know anything about Broker B (even though the topics are identical)

So what we are getting at is that we BELIEVE your old Sonoff devices are configured to talk to the old MQTT broker on your old system - if that is the case then they will never send or receive messages to the new broker on the new system - anything on the new System listening for messages from those switches will never see the messages - unless they are configured to talk to the IP address of the old System.

None of us know the tutorial that you followed so we need to know what firmware was loaded on the Sonoff Switches - you should have a form of login page to them that will show this information together with some information in relation to the configuration of the MQTT system that the switches are configured to talk to

Craig

Actually the timestamp you inject is shown in the debug pane under debug XX, after it has passed through the MQTT broker.
"2026-03-19T20:16:01.921Z" is in UTC string format, which suggests that the mqtt-in node is set to output a string. Usually I keep the default, which is "auto detect".

The second of your screenshots of the log show Node-red connecting to the broker on localhost.
That does not confirm that Mosquitto accepts connections from beyond localhost. For that you should be looking for other IPs than 127.0.0.1, as in my logfile examples.

I think we need to know the IP of your new Pi and broker and the IP that the Sonoff devices are configured to connect to. It should of course be the same.

I appreciate the fact you folks even look at this thing so let me try to clear the air here. The so called old system was created over two years ago and is still running today. It is a RPi3B running node red and mosquitto whose versions were current at that time. The sonoff switches were loaded with alternate firmware that included the i/p address of the broker running on the same RPi as the NodeRed. This alternate firmware sends the state message (on or off) back to the broker which in turn sends it to the NR dashboard as well as the debug pane. All these devices are running on the same in-home network. The alternate firmware came from the RNTlab and not from HomeAssist or Home Automation or Tasmoto.
The "new" system is an RPi4B running Trixie, the most current NodeRed (4.1.7) and the most current Mosquitto (v5) . The new system is also using the same i/p address as the old system and the two systems are NEVER powered on at the same time for obvious reasons. All of the screenshots that I am showing you, were taken from the new system. The flow diagram that I included was for simply troubleshooting this issue. I realize that the simple four node flow apparently does not prove that the message was exchanged via wifi . In fact, I still don't understand how "timestamp" gets from the "MQTTout node to the MQTTin node"?
The old system contains 8 flows identical to the switch node at the top of the diagram all with unique topics. When I built this new system, I exported the FLOWS from the old system and then imported them into the new system. The whole system deployed with no issues (much to my surprise) but it did not work leading directly to where I am right now. One big difference in the new system is the fact that it is password protected as evidenced by the use of this line in the mosquitto.conf file ; "allow_anonymous false". I tried once to set that to "true" and see if that made any difference but no change.

I hope this helps to clear the air.

Let's see if we can demystify that, or if not, possibly overwhelm you with too much information...

The mqtt-out node in Node-red connects to the Mosquitto broker.

There are many kinds of traffic possible over a network connection and they are kept apart by means of the "port number".
Website traffic uses port 80, Node-red uses port 1880 & MQTT uses port 1883.

When you click to inject, the timestamp gets sent out by mqtt-out over the "network connection" to the broker's IP address.
However your Pi's networking software is smart enough to know that the broker's address 127.0.0.1 means "this machine" so the message does not go to your router, instead it is given to whatever software on this Pi is "listening" to port number 1883 - Mosquitto.

The message has a topic - "test1", which basically says what the message is about.
You could use a topic such as "sonoff/1/status" to indicate what it's about (status) and where it came from (sonoff/1).
And it has a payload, perhaps a timestamp, perhaps "ON", but it could be anything.

What does the broker do with it?
First it looks to see if anyone has told it that this topic is of interest, by subscribing.
Your mqtt-in node has subscribed to the topic "test1". Now any message with that topic, from any source will be sent to the mqtt-in node and to anyone else that is subscribed to that topic.
Once it has sent the message to all the subscribers, Mosquitto discards the message.
Obviously there is more going on than that, and there are options such as retained messages, last will and testament messages etc. You will know all about these from the MQTT videos at hivemq.

The mqtt-in node is subscribed to the topic "test1". That means "I want to get a copy of every message with the topic 'test1' ".
It receives the message and passes it on down the wire to the debug node.

Your Sonoffs have an IP address on the network.
It might be 192.168.1.nnn or 10.0.0.nnn

When they send a message for the broker, it does go via your router, which recognises that the target IP address is on the home network and passes it to that device - the Pi.

But:

  1. Mosquitto, by default will not accept connections from any IP address other than 127.0.0.1. It has to be correctly set up to accept external connections.
  2. Your mqtt-in node has to subscribe to the same topic as the Sonoff publishes to, because if nobody is subscribed to the topic, Mosquitto discards the message.

So far you have not shown us any evidence that these conditions are satisfied.

1 Is Mosquitto accepting external connections?
What I did to reduce the size of the logfile was a) sudo rm /var/log/mosquitto/mosquitto.log and b) sudo reboot.
When the Pi rebooted the file was nice and short & I could see successful and failed connections by means of sudo cat /var/log/mosquitto/mosquitto.log or by opening it with an editor. It might take a minute or two for the Sonoff to reconnect.

2 If the Sonoff connects, you should start to see in the log when it publishes status messages. No idea how frequent these are, unfortunately, or even if a Sonoff does issue such messages.
What topic is it publishing to? Your mqtt-in has to subscribe to the same topic (you can use wildcards though, so eg sonoff/+/status would subscribe to sonoff/1/status as well as sonoff/northpole/status)

The Sonoff also subscribes to a topic, which might be sonoff/northpole/command
If you want to switch it on, Node-red mqtt-out might have to send "ON" to this topic.

Those topics and payloads are all made up, I don't know what you really need to send, but your old Pi setup will have the details.

In reading through all of this I would ask why don't you install a mqtt explorer and see what messages are going where?

There is no “Mosquitto v5”.
You’re probably thinking of MQTT v5, which is supported by Eclipse Mosquitto version 1.6+ (and all 2.x releases).

On my RPI-4B I have... mosquitto version 2.0.11 running.

As Gerry suggested, download/install MQTT Explorer as you may then see what is going on.

I 3rd (or 4th?) the use of MQTT Explorer, it is a must for understanding your MQTT broker, topics and message flow.

The first thing that you will discover is whether your new broker is accepting external connections - assuming MQTT Explorer is run from a laptop/desktop or other device that isn't the Pi.

If you can connect, you will then see ALL of the messages that the broker is handling. When you set up a connection in MQTT Explorer, if you click on the "Advanced" button, you should see the following:

This shows you the default MQTT topics that it is subscribing to. # is a wildcard and means "All standard topics". $SYS/# is also a wildcard but for the broker's special internal $SYS topic structure that normally isn't seen but is useful for monitoring the broker's performance.

You will be able to see as topics are updated and you will build up a history view of topics as well as long as the explorer is running.

When you first start it, you will only see RETAINED messages on the broker until a device updates a topic. (and the $SYS message topics of course).

Once again, thanks for the help. And it is help for me.
The system does have an i/p address 192.168.1.151 and I am assuming the is the same as "localhost pr 127.0.0.1" which I am using.
for "dynamicdave" .The v5 came from here;


There is also a reference to v5 in the screenshot of the MQTT out node where I selected v3.1.1 . If you pull down the protocol button V5 is a choice. But I can not take a picture of it.

I have MQTT Explorer installed on my pc and I going there right now and see if I can work my through the operation. I have also added a couple more shots of the flow.


![MOSQ_8|582x500]

Also note that both MQTT in /out have the user/pswd set on the security tabs.

Thanks.
Your screenshot is saying you have installed Mosquitto version 2.0.31 and that supports MQTT protocols v5.0/v3.1.1 and v3.1

On all of my RPi servers I'm using v3.1.1 protocol (I'll have to experiment with v5.0 in the future).

As other people have said - change localhost to the IP of your RPi that is running Mosquitto.

Unless you are exposing your RPi to the Internet, I'd avoid setting a user/pwd for now.
Keep it simple. Get your set-up working then take the next step (if needed).

Here's an example of the setting for one of my RPi(s).


Note: If you remove user/passwd, you will need to adjust Mosquitto's conf settings.