/var/log/boot.log
syslog didn't match 10.8.0, so I did:
raspberrypi[2022-10-27 09:30:48]:/var/log>grep 10.8.0 *.log
boot.log: My IP address is 10.8.0.1
I did a dmesg -e and I see one thing that maybe is a clue?
[Oct27 03:00] TCP: request_sock_TCP: Possible SYN flooding on port 1883. Sending cookies. Check SNMP counters.
I don't know what SNMP counters are, or what communicates on 1883.
Looks like it might be mosquitto (MQTT). I'd set that up once to try it out, but I never ended up actually using it. I don't know what could have changed WRT that, but I have had other seemingly unrelated issues recently, but maybe they're related. I'd set up a "new" wemo motion sensor, and even though it gets through setup, the device never appeared in the wemo app's devices tab (but does show up in the node-red-node-wemo node config. Another issue is that my custom pi-bose code (a fork of pi-rc) seems to run into memory allocation errors much more frequently than it used to. The source code probably has a memory leak affecting the stack, but it used to go for like half a year before it started happening. It has happened recently within 24 hours a few times in a row.
That's part of what made me think my pi was just dying... But really, I have no idea.
Let me try uninstalling mosquitto and see what happens...
Well apparently, I previously uninstalled mosquitto. I did check the tasmota config and it still had MQTT enabled. I disabled it on both outlets.
I ended up sending my UDM support logs to Ubiquiti and they're going to analyze them and follow up over email. It seems there had been a UnifiOS update this past Monday.
I need to set this aside and do actual work, so I'll have to come back to debugging this for now.
Ah, you are running an older version of the OS than I am.
Have a look at the lines around the My IP line.
Also, to see what process is using port 1883 run
sudo netstat -anpe | grep "1883" | grep "LISTEN"
It's the pilight-daemon.
boot.log-[ OK ] Started Login Service.
boot.log-[ OK ] Started Load/Save RF Kill Switch Status.
boot.log-[ OK ] Started netfilter persistent configuration.
boot.log-[ OK ] Reached target Network (Pre).
boot.log- Starting Raise network interfaces...
boot.log-[ OK ] Started Raise network interfaces.
boot.log-[ OK ] Reached target Network.
boot.log- Starting Permit User Sessions...
boot.log- Starting /etc/rc.local Compatibility...
boot.log- Starting OpenBSD Secure Shell server...
boot.log: My IP address is 10.8.0.1
boot.log-Starting OpenVPN service...
boot.log-[ OK ] Started Unattended Upgrades Shutdown.
boot.log-[ OK ] Started /etc/rc.local Compatibility.
boot.log-[ OK ] Started OpenVPN service.
boot.log-[ OK ] Started Permit User Sessions.
boot.log- Starting Light Display Manager...
boot.log- Starting Terminate Plymouth Boot Screen...
boot.log- Starting Hold until boot process finishes up...
boot.log-[ OK ] Started LSB: Autogenerate and use a swap file.
So you have got openvpn running.
Run
sudo systemctl status openvpn
and if that says it is there and running, then
sudo systemctl stop openvpn
and to stop it running again at boot
sudo systemctl disable openvpn
Then check that the tun0 interface is no more.
Did you know you were running that, whatever it is?
Have you opened any ports to the internet so you can access the pi remotely, for example?
OK, well this shows how familiar I am with VPN (not very). I thought my install of openvpn was to be able to connect to my VPN provider. I'll have to check my notes to see what I might have been doing when I installed openvpn... But yes, it is running.
And I use pilight every day. I have 13 cheap RF outlets and a breadboard connected to the pi set up with a 433Mhz transmitter and receiver chip and antenna. I disabled the receiver because pilight's processing of background RF was occasionally overheating my pi, so I only use it to transmit commands to the RF outlets, not catch changes using the remotes they come with.
And no, there are no explicitly opened ports. The only thing I have that reaches out is an ssh tunnel to webhookrelay so that I can make remote webhook calls from my iPhone.
My UDM has a VPN server that I use when I want to connect to my home network remotely. Incidentally, it doesn't support mDNS (or perhaps I just haven't configured that).
I'll need to consult my notes to see if I'm actually using that openvpn service to be sure I'm not breaking something by disabling it. I definitely have overloaded this pi. I have an unopened pi4 that I've been meaning to set up, but I have so many projects to tackle before that lol...
Run top just to make sure that there is nothing hogging the processor or using up all the memory. That sort of thing can cause apparently random strange behaviour.
Good suggestion. The top process is homebridge ranging from 10-25% CPU. tails off dramatically after that. Node red is second at around 7%.
And the memory? What do you see at the top of top page?
top - 12:09:37 up 19:43, 2 users, load average: 0.99, 1.01, 1.15
Tasks: 120 total, 1 running, 67 sleeping, 0 stopped, 0 zombie
%Cpu(s): 10.1 us, 27.6 sy, 0.0 ni, 62.3 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem : 879284 total, 33708 free, 323380 used, 522196 buff/cache
KiB Swap: 102396 total, 101884 free, 512 used. 498540 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12387 pi 20 0 240228 135616 35048 S 3.6 15.4 62:35.27 node-red
1141 pi 20 0 214912 108956 34156 S 23.7 12.4 188:26.33 homebridge
830 pi 20 0 190076 81260 34052 S 0.0 9.2 2:46.30 hb-service
No problems there. Not at that moment anyway.
Agreed, though it's during those conditions when I get the "Connect failed" message in tasmota emanating from the WebSend call. And I suspect that was the case when I was testing in the browser yesterday too, though I didn't check it at that time. Point is, all other functions of the Pi seemed to work as expected.
I have a number of other webhook calls to the pi set up. Let me try some of them and see if they also fail, and if I get a more detailed error...
I'm disappointed that I haven't heard back from Ubiquiti yet. Hopefully I'll have some time to dig into this again this weekend. It has to be the Unifi os update. Webhooks just suddenly stop working on 2 devices... it's too coincidental.
I guess one thing I can try is a tasmota firmware update...
I just poured through my unifiOS logs. Can't find a damned thing. Lots of occurrences of both Sonoffs, but nothing that indicates a problem. I learned that I can issue the WebSend commands manually from the Sonoffs in tasmota and I tried doing it with a tail -f on both the node red logs and with various system logs... Nothing's hitting the Pi. It must be the UnifiOS update, but I cannot find anything useful in its logs. So frustrating.
You may have to run something like tcpdump to check the requests are getting to pi and if so why they are getting blocked.
