This one's new for me. Today I suddenly was unable to log into one of my Pi's with VNC, which I always use for everything. I was going to take an image backup to have something current. But VNC didn't log in. So I tried to log in with Node-RED to reboot it, but that didn't work. I tried Home Assistant on the same PI, and that worked without a hitch. But the reboot didn't change anything, same thing.
Now the weird thing is that even if I can't log in to Node-RED's GUI with a browser, the MQTT messages keep coming from the RFXtrx and Tellstick Duo that are connected to the thing. Has anybody seen something like that, and can you please tell me how to fix it?
I'm on my cabin, so if it can't be done over any remote, I'll just have to drive over there and plug in the backup MicroSD that I have taped to all my Pi's. That's of course older than what I had, but I'm pretty sure I can simply copy the latest flow from the old card to the new.
Maybe it's not SSH? I use Putty and try to log into port 22. But it says "connection type SSH". And Putty says: "Network error: Conection refused" in a small popup-window with the title bar "Putty Fatal Error". Nothing more.
"Curiouser and curiouser, cried Mastiff." I can actually SSH in from another Pi, it just doesn't work from Putty. So...is there anything I can do to try to fix this on SSH? I am pretty much lost without an GUI like VNC...
So after typing "vncserver" from SSH I can get into the thing on VNC, but all I see is "cannot currently show desktop". I guess I'm screwed and need to go to the backup, right?
I solved it a few minutes ago, thanks! Turned out that the disk was totally full. A script had saved a bunch of smaller backups to the SSD itself, and then some more files had come along and filled the SSD up (I only use 16 gig of the 64 gig USB SSD because I want to be able to use my 16 gig MicroSD-cards as a backup). I used ncdu to drill down and find out where the backup was hiding and killed it. All is well again!
Edit: The reason is that even if I have the actuall, correct storage space for backups, which is a Samba share, in fstab, it sometimes doesn't mount that automatically. So the backups were saved locally as a fallback, and that filled the SSD.
2nd time in a few days that someone has mentioned full disks.
I strongly recommend that people set up Node-RED to periodically check for low disk space and send an alert over your favourite messaging tool (Telegram for me). Also worth setting up a similar check for increasing utilisation of your swap space since that indicates something chewing up memory.
That sounds rather sensible. This actually happened to me twice today! Because the PI's had been running for exactly as long and had the same backups. So I guess it should be put into a flow. Another thing is if you run Home Assistant on the same Pi. If there's a lot going on in the network the db-file can in a few months be several gigabytes.
Turned out that the disk was totally full. A script had saved a bunch of smaller backups to the SSD itself, and then some more files had come along and filled the SSD up (I only use 16 gig of the 64 gig USB SSD because I want to be able to use my 16 gig MicroSD-cards as a backup). I used ncdu to drill down and find out where the backup was hiding and killed it. All is well again! 