Unexpected flow restart with lost of context files & tokens (google and telegram)!

Node-red 4.1.2

My flow randomly restart (each 2 or 3 days) and then all the context files are deleted (the variables are not re-written).

Furtherlore the tokens field for telegram and google are blank.

Is there a log file that could provide an indication why the flows restart?

Remark: If I restart the flows by myself everything goes fine; the context variables are correctly initialized and the tokens not lost

Welcome you the forum.
We need to see what comes before the restart.

It is probably in /var/log/syslog

There is no syslog in /var/log but I can read the log with journalctl command.

I don’t see anything that can explain the restart of the flows (see the screenshot in my initial request).

Actually I suspect node-red-contrib-googlehome to be the root cause. I have just deleted all the nodes relying on this palette and wait if the problem comes again

Find the node red start and see what comes before it.

Looking at the journalctl trace, nothing special seems to happen before the flows stop.

In future please copy/paste logs, not screenshot.

image

That is the result of a Deploy. If it wasn't you then, if your system is open to the internet, then you have been hacked. Disconnect the system from the internet and stop node-red.

1 Like

image
This looks to me like evidence of malicious cron jobs.
Unless you intend to fetch and execute bash scripts from that site, I too think you have been hacked.

1 Like

Yes, you are right. I missed that. Unless @sebamelo is doing something rather unusual then his machine has been well and truly hacked. Not just messing with node-red but also adding cron jobs.

@sebamelo I think you need to recreate the SD card on the pi and start again. Hopefully you have got a backup of your flows from before the hack. Also you need to consider the possibility that other machines on your network may have been infiltrated from the pi.

See this FAQ post for advice on how to safely access node-red over the internet.

@sebamelo will you please tell us if your Node-red machine has ever been accessible from the internet (by you or anyone)?

If it has been, how did you set it up?
Does the user raspberrypi have the right to run sudo, with or without a password?

The hacker has made multiple changes to your computer, not confined to Node-red. You will have to reinstall the entire operating system from scratch.

The download site seems (in chatgpt's opinion) to be on Cloudflare. Perhaps they would be interested to know they are serving cryptomining code for hackers.

Hi to all,

Yes, my raspberry is accessible from internet by means of NAT.

I forward one specific port of my internet box public address to the port 1880 of my raspberry private address (192.168.1.48).

There is only one single user (pi) with admin privilege but protected with password.

FYI I used Raspberry PI image utility to download the operating system

It seems it is not really safe!

Can you recommand me a safer site?

It takes less than 30 seconds for a newly exposed address on the Internet to start being probed by automated attacks.

You should NEVER expose anything unless you really know how to secure things. Node-RED was not designed to be especially secure and, in my personal opinion, both should not be considered that way, nor should anyone try to make it so. Cyber security is a specialist task that requires multiple layers.

That said, there certainly are ways to open up some part and to make it secure even if you are not an expert. I've been saying this for years now but I will repeat it again. Unless you are an expert in Cyber security, choose a trusted partner to provide the security. They will have spent a lot of money on getting things right for their customers.

Right now, there is probably nobody who does a better job at keeping sites safe than Cloudflare. Their secure proxy service is called Cloudflare Zero Trust and I did a write up on how to get it set up recently. Their free tier is more than enough for home use. Not only does it provide the secure proxy gateway but also up to 50 free users for logins.

Remove the open port from your router, build a new OS image from trusted source. Carefully go through your flow backup and make sure you only implement the things that you understand and are yours. Set up CFZT or some other tool to provide a secure proxy without having to open ports on your router. Make sure that you use external logins and not Node-RED's login.

If that is all too much, maybe consider using FlowFuse.

1 Like

Thank you for your advice; I will competely reflash my raspberry.

Anyhow only port 1880 was exposed and I haven’t noticed any abnormal connection to Node red (looking at node-red-log) neither to the raspberry OS (looking at journalctl).

Why hacker would waste time to control a single raspberry?

Thanks again to have replied to my issue.

There are a LOT of under-employed people in adversarial countries with too much time and too much Internet bandwidth I'm afraid. Though a lot of attacks are purely automated anyway, at least in part.

There is money to be made by adding compute and bandwidth to botnets, botnets may control advertising usage scams, social media misinformation campaigns, denial of service attacks and much more. For some social media topics, 70-90% of engagement is now coming from bots.

Each Pi added may not net a lot of cash on its own but multiply by 100's if not 1,000's, it starts to add up, especially for folk with few other options for income and low cost of living.

image

Did you do a deploy at that time? If not then someone else did, that is definitely abnormal.

Was the cron job that @jbudd pointed out from the log, which downloaded a script and ran it, abnormal. Once into node red a hacker can easily create a cron job.

That is most probably not how you got hacked, it is from opening node-red to the internet.
As @TotallyInformation said, a bot will find your system probably within less than a minute and, if you have password access enabled then it will start bombarding your system with login attempts to try and get in. Read the FAQ that you were pointed to earlier and pay attention to what @TotallyInformation says, he knows what he is talking about.

1 Like

Port 1880 is a well known way to gain access to a poorly protected system, automated port scanners will very quickly detect opening up this port.
You could have forwarded external port 65302 to internal port 1880, which would have made the open port much less noticeable. (But don't do this, once a port scanner has seen it, you are equally vulnerable.)

Once the hacker got access, probably through Node-red, they set up persistent cron jobs to download and install whatever malicious software they chose.
They only needed access once.
From then on, all traffic was initiated from the Pi.
The example I looked at installed a crypto-miner on your Pi.

This seems to have been a professional hacking organisation in China.
They did not target just your Pi, they target every computer in the world which has port 1880 open.
No doubt they use other attack vectors too.
I doubt if your Pi can generate very much income for them from crypto mining, but it's your electricity they are wasting, so why not?

2 Likes

Like many other Linux distributions, Raspberry Pi OS is set up more for convenience than security.
Particularly troublesome is that the standard user has the right to use sudo without entering a password, and setup cron jobs.
Combine that with port forwarding and Node-red having access to operating system commands via exec and you have a very vulnerable combination.

Never use port forwarding!

1 Like

Is that still true on the Lite version? I thought it had been changed.

Any user can setup cron jobs that are run by that user, so user cron-jobs can be setup from within node-red without having to use sudo.

I reinstalled RPiOS 64 bit Trixie lite on a Pi Zero 2 yesterday and it definitely let me do sudo systemctl enable nodered.
There are some changes specific to the lite version for Trixie, for example it's almost impossible to change the hostname.

Yes indeed.
We don't know what Node-red flow was used to commit this specific hack, but it may well have been an exec command.
I don't think it's possible to prevent Node-red from accessing non-privileged commands so I'm wondering if securing Node-red should include adding the user to /etc/cron.deny.
I don't know if that would break the inject node and contrib nodes such as cron-plus

Can't you just change it in /etc/hosts and /etc/hostname?