I've been running a node-red flow that take data from a raspberry pi and inputs it into an online postgres database. It's been running fine fr over a year. Then ealier this week I was notified of a data stoppage and when
i opened node-red my flow was gone. I restored a backup from a few days prior to the stoppage and it went back to working great again. Then this morning I had another notification of stoppage, I check node-red and sure enough, flow is gone.
I am at a loss as to where to begin to search for a cause. Any help would be greatly appreciated.
First thing to do is to do a full filing system check (assuming you are running from an SD-Card). Also check the system logs and DMESG to see if any hardware errors are showing up.
It goes without saying that no computer system is perfect which is why backups were invented, make sure you have important data of all kinds backed up to at least 2 other locations, one of which should be offsite.
I am not running off an SD-card. I am running from a cloud host (Digital Ocean).
OK, still possible to get a corrupted FS but somewhat less likely - worth checking anyway.
Does the problem occur at a set time/day?
Have you set Node-RED to run with its own user/group? And if so, have you locked down the userDir folders to only that user/group?
Are you copying the systemd log to syslog and if so, can you see any errors before the problem happens?
And how well protected is your Node-RED setup and your server? Are you sure that you don't have an unlwelcome "visitor"?
I am very much a lightweight when it comes to the inner workings as I pretty much just followed along with some tutorials to setup node-red from the start.
I will dive into things deeper and look into the user/group and logs, and you did make me wonder about the time of day thing. It didn't happen at exactly the same time but it was within 30 minutes so I may just do another restore and see if it happens again around that time frame and that may help narrow the hunt.
Thanks for taking the time to throw some ideas at me.
Hi, this also happend too me, IĀ“m using Digital Ocean and yesterday all of my flows where gone.
Restored from backup, but one hour later the flows where gone again.
I ended my server, and will try later i the future to set it up again.
I seem to remember reports of losing data when the droplet restarts. Could it be that?
Wow, that's interesting. I guess I will contact Digital Ocean's support and quiz them about it. Thanks for that bit of info.
I have an update...
I restored another backup and tried to keep a close watch to see what happens, and after a few hours of running fine a window popped up with:
"The flows on the server have been updated" with options to Ignore or Review Changes.
I click review and get:
"The server is running a more recent set of flows. The changes include no conflicts and can be merged automatically"
After which my flows vanish.
During this time there was no updating of anything by me to anything on the server.
Does any of that help anyone point to a possible cause of this?
Oh, and I did contact support at Digital Ocean and they basically said it's not their problem and directed me to this forum! I may be shopping for a new online cloud provider.
The most obvious answer is that the flows are open in another editor somewhere and are being deleted by someone
Since that someone isn't you - maybe time for some password changes?
1 Like
Seems that we have seen several instances of this related to Digital Ocean, though they are normally considered a very good host.
I assume you went through the previous recommendations and checked for filing system errors?
When you get the message saying that the flows were updated, don't do anything in Node-RED, go to a command line and manually check if the flow file is there or not. If it is there, find out the user ID that made the change. Also, if it is there, make a manual copy so that it can be separately reviewed. Check that copy for unexpected entries.
Also, double-check that nobody else can get into the editor by turning on Node-RED's login. See if that fixes the "issue".
What you are doing here is checking for a 3rd-party messing with things. And yes, change the passwords and reboot the server. Also make sure that you have turned on the virtual server's firewall and check for logins that don't come from you. Maybe restrict logins to your own country.
Is Node-RED protected from external access? Does your server expose SSH on port 22 and does it have secure enough passwords?
Was your node-red open to the internet at any point, without strong protection against hackers getting in?
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.
