Need help rebuilding home automation raspberry pi

Hello, I managed to get things going again. I found that the RPi had decided to change it's i/p address. I had to run nmtui again to get the i/p back where it belonged. And then I had to review what we did before. so the first set of numbers came from the log where the line "password_file /etc/mosquitto/passwd" had been commented out of the .conf file

ckentm@4004HOME:~ $ sudo cat /var/log/mosquitto/mosquitto.log
1774647740: mosquitto version 2.0.21 starting
1774647740: Config loaded from /etc/mosquitto/mosquitto.conf.
1774647740: Opening ipv4 listen socket on port 1883.
1774647740: Opening ipv6 listen socket on port 1883.
1774647740: mosquitto version 2.0.21 running
1774647747: New connection from 127.0.0.1:35498 on port 1883.
1774647747: Client nodered5a9c568a1924676e disconnected, not authorised.
1774647762: New connection from 127.0.0.1:59148 on port 1883.
1774647762: Client nodered5a9c568a1924676e disconnected, not authorised.
1774647777: New connection from 127.0.0.1:58904 on port 1883.

And the next numbers are from the log that was taken after the line was added back into .conf file.
ckentm@4004HOME:~ $ sudo cat /var/log/mosquitto/mosquitto.log
1774649340: mosquitto version 2.0.21 starting
1774649340: Config loaded from /etc/mosquitto/mosquitto.conf.
1774649340: Opening ipv4 listen socket on port 1883.
1774649340: Opening ipv6 listen socket on port 1883.
1774649340: mosquitto version 2.0.21 running
1774649344: New connection from 192.168.1.221:51650 on port 1883.
1774649344: Client 4004_05 disconnected, not authorised.
1774649349: New connection from 192.168.1.221:59720 on port 1883.
1774649349: Client 4004_05 disconnected, not authorised.
1774649354: New connection from 192.168.1.221:50706 on port 1883.
1774649354: Client 4004_05 disconnected, not authorised.

So it appears that there is some difference in the output of the log file. Where that line commented we get this message
"New connection from 127.0.0.1:35498 on port 1883."

But when the line is present, we get.
"New connection from 192.168.1.221:50706 on port 1883."
We seem to be connecting to the sonoff at that i/p address.

I hope this can help you. I, at least get the idea that the msg is getting to the switch although I do not hear it or see it operating.

Let me know what you think
Charles

It is a very good idea that a headless Pi always retains the same address.
It is essential for a Mosquitto server to always have the same IP address.
By far the best way in a home network to achieve this is by reserving the address in your router web admin interface. Different routers need different approaches but you can hopefully find "address reservation" somewhere in the setup.
For this you will need to find the MAC address of the Pi. It should be available with cat /sys/class/net/eth0/address (ethernet connection) or cat /sys/class/net/wlan0/address (wifi)

IP address 127.0.0.1 AKA localhost is the Raspberry Pi itself. This is Node-red trying to connect.
I cannot comprehend what circumstances might cause connection not authorised for localhost if allow_anonymous is true.

If I was experiencing this, I would delete the broker config in Node-red and make a new one using all default values except name localhost and server localhost

Or even reinstall the OS, Node-red and Mosquitto.
Make sure the Pi has a unique hostname (set in Raspberry Pi Imager) and a reserved IP address.
Verify that you are connecting to the right Pi by means of the hostname command

Unfortunately I don't share your optimism. You are doing something fundamentally wrong.

Please Export one of your mqtt nodes and paste it online here. Also tell us the ip address of the pi.

Just in case there is some strange incompatibility between the latest versions, I spent part of this morning on a complete reinstallation and test with a Raspberry Pi Zero 2 W (which is all that's available, and which implies connection by wifi)

Burn RPiOS 64 bit lite to SD card (click the arrow to expand)

Use Raspberry Pi Imager to burn RPiOS 64 bit lite to SD card for Pi Zero 2 W.
hostname: greenpi
localisation: London (United Kingdom)
username: pi, password Password1
WiFi: My 2.4GHz wifi
ssh: enabled, password authentication
RPiConnect: disabled

Boot Pi

Leave 5 minutes for setup before it appears on the wifi.
The IP address is 192.168.1.28, which was previously reserved in the router for this hardware.
ssh pi@192.168.1.28
Connect over ssh
Make sure I'm connected to the right Pi: hostname shows "greenpi"
sudo apt update && sudo apt -y full-upgrade
sudo reboot
Connect again

Install Node-red

bash <(curl -sL https://github.com/node-red/linux-installers/releases/latest/download/update-nodejs-and-nodered-deb)
Install Pi specific nodes: No
Accepted defaults in the security setup after installation.
User name admin, password Password1

sudo systemctl enable nodered
sudo systemctl start nodered
Point a browser to 192.168.1.28:1880 to verify Node-red is running.

Install Mosquitto

sudo apt install -y mosquitto mosquitto-clients
sudo systemctl enable mosquitto
Create file /etc/mosquitto/conf.d/my.conf :
printf "%s\n%s\n" "listener 1883" "allow_anonymous true" | sudo tee /etc/mosquitto/conf.d/my.conf

sudo reboot.

Test that it works

Test access from Node-red on the new system (localhost)

Browser points to 192.168.1.28:1880 Pull inject, mqtt-out, mqtt-in and debug nodes onto flow. Wire them up. Add a broker config. The only fields I changed were on Connection tab: name localhost server localhost

Topic: test
Deploy

The mqtt node statuses showed "connected"
Inject. Timestamp is output to debug.

Test access from somewhere other than localhost.

I tested with Node-red on a different Pi (The browser runs on my PC), though I could have used any device on the network, as long as I could control it's MQTT setup. Add a new broker, name greenpi, server 192.168.1.28 Same test as above: timestamp does show up on the debug pane.

That's a complete account of what I did, I did not install something else that I always use so forgot to mention it :grinning_face:

It works for me.
What did you do different from the above?

I did the same on a PI 4B using trixie, using the mosquitto instructions I posted, everything works, I can't help but think this is an IP thing somehow, firewall, something, I wonder if he did a thorough network analysis or maybe even just a simple ping test? Or a command to see if the port is open? I can see where this is frustrating though.

Another possibility might be isolation by the router, if that's the word, of ethernet and wireless devices.

We really don't have a clear enough view of the OP's setup

That would still be an IP thing. You are right though we should get a network topology as that's the most basic level to start I think.

Gents: please stand down while I try to get some info together. At this point my procedures are exact duplications of yours. The install for this whole thing comes from the RNTLAB.com manual "Smarthome_RaspberryPi etc." v1.7 which is the latest and matches your procedures exactly. The RPi4 does have a static i/p that I set after the RPi is booted the first time using the nmtui utility. I have discovered the procedure for assigning an i/p reservation on my new router (an eero pro7) provided by my ISP. The drawback to that is the fact that I have to use my phone to access it. I am just not a phone person. Another thing I would like to point out is that my RPi is connected to the network via eth and I have wondered if I even need to provide wifi credentials to the RPi?
The really flabbergasting thing about this whole situation is the fact that sitting right here on the same bench is an RPi three running the same program albeit with earlier versions of software. It is connected to the same network and Yes, it has a different i/p than this new system and it runs things in the house daily with no issues.
So let me get some information together. I will take screenshots of the mqtt node configuration both in and out. Also note that the simple four node test works fine in this new setup producing the timestamp into the Debug pane every time. And the two mqtt nodes show they are connected.
I greatly appreciate all the help from you folks fully realizing the time and effort you have expended. My intention at this point are to rebuild from scratch.
Regards
Charles

Does Mosquitto default to local host only? Will mosquitto not talk to devices on the LAN network?

Welcome to the forum @ab9nq

Yes it does.

It will do if it's correctly set up and if there isn't something else blocking the connection.

Yes, since version 2.0

It will, that is what the line
listener 1883
in the config file does, it tells it to allow connections from outside the localhost.

Also it defaults to requiring user and password login, the line
allow_anonymous true
tells it to allow connections without user/pwd.

Which is it listen l883 for local host only -no LAN connections
or

listen 1883 0.0.0.0 --LAN connections?

and

allow_anonymous true  --less secure but simplest.

It is listener, not listen.
listener 1883 allows access from anywhere. There are addional options such as bind_address that allow more control. See mosquitto.conf man page | Eclipse Mosquitto

No, export the nodes and paste them here.

If it is connected via ethernet then you can ignore wifi. Be careful though, if you configure wifi too then the pi will probably have two different ip addresses, one for the ethernet and one for the wifi. Make sure you are setting the right one. The command ip a will show you the interfaces and ip addresses so you can check it is all what you expect.

Chuck it in the bin and get a decent router with a browser i/f.

Your ISP provided a £300 router? Who are they?

I think the eero needs to be connected to an external ADSL/cable/etc modem, so presumably one of it's two ethernet ports is occupied by the modem cable.
Is the Pi plugged directly into the other port or do you have another switch or router in between?

@jbudd has a good point. Can you ping the new pi from the old one?

Apologize for my confusion. Thank you for the link.

I am getting ready to rebuild again but here are answers to the latest questions.

  1. I can ping the new RPi from the old RPi.
  2. The 2nd port of the router goes to a 2.5G switch which goes to a 12port 1G switch and then all over the house. The two pi devices are plugged into that switch.
  3. The link to the flows of the new pi are;
    flows exported this moring

Charles