Just thought I'd share a little experience I had yesterday regarding a Crypto Miner I discovered inside my Node-Red. I have Node-Red in a Docker Container and noticed my server CPU usage spiked to nearly 100% full-time.
Upon researching the cause, I saw the responsible process was /var/tmp/sustes -c /var/tmp/wc.conf
I did not find this on my root file system.....so I started inspecting my Docker Containers one-by-one until I found it in the Node-Red docker container. So THEN, I opened up the Node-Red Web UI and found a whole bunch of these subflows inside my Node-Red!! Anyway, just thought I'd share
This is highly likely my own problem and not widespread......but I wanted to make people aware. This is a thing that is happening now. So watch out.
I found more info about the miner from a Digital Ocean post when I searched for "sustes" along with how to get rid of it. In addition to deleting the files responsible, I obviously had to delete all the subflows which had crap in them.
Just to add a key missing piece of information here .. this was a node-red instance exposed on the internet with no security applied.
This is why we always go to great lengths to highlight the security implications of exposing NR to the internet. The exec node gives whoever can deploy a flow complete access to the machine running NR. Do not assume that no-one will find your NR instance - this demonstrates NR is now recognised and targeted by people scanning the internet for vulnerabilities.
If you really must expose it to the internet, at the very least ensure you have adminAuth configured so only you can log in and see the editor.
I did in fact have Node-Red under a Reverse Proxy (NGINX) with BasicAuth user/password. Unfortunately, I did NOT close the Port for Node-Red.....so that is definitely a wide-open attack vector (stupid me). This is why I said "highly likely my own problem and not widespread".
Outside of that, it is only a guess as to how else my instance could have been compromised.
Node-Red subreddit users had asked me to post this here in the official forum to make more users aware that Node Red definitely is a well-known target to attack now. For my personal experience, I am very thankful I had Node-Red in a Docker Container Environment. The issue was self-contained and could not spread.
That is certainly the part that is interesting to me. That there are bad actors out there with Node-RED knowledge!
Long timers will know that security is something that I bang on about. Hopefully this is another wakeup call. I recognise that this was an accident but we all need to remember that, no matter how good Node-RED is (and it IS good!), it is not security hardened so safely exposing it to the Internet is never going to be all that simple.
A few thoughts from me on the matter are here if you are interested.
Thanks for sharing this with us.
Haha, just realised that it is weird that the perpetrator added debug output and even left it on!
In addition to configuring adminAuth, I have also configured open VPN and only opened the port on my home LAN router for vpn access ... so this properly secures my home LAN from the outside world while I can still access it from my smart phone and laptop anywhere in the world.
Yes, that helps when you are running on-site. But this was running on a cloud server. I'm not sure what DigitalOcean's rules are on running a VPN, many hosts don't allow it. My own VPS's run on BHost which does permit VPN's as long as they are restricted.
I'm on a Hetzner physical server......That Digital Ocean post was just something I found on the internet to help me diagnose and treat my problem. I am not using anything from Digital Ocean.
OK, well if that is on a local network you control then setting up a VPN may be useful. Too many variables to say for sure. For example, we don't know if you are on a corporate network or a home network.
Could well be. Or it could be that some bored script kiddie in East Europe or elsewhere has discovered Node-RED and is searching for accessible systems and other evidence of Node-RED using a search engine.
That is very possible. Interestingly, there were like 20 or so identical subflows like the screenshot I posted.....Each subflow was named differently with just like five to eight random characters.
Update: Oh an also....my Node-Red instance was not on my Hetzner European server (apologies)......it is hosted on my home server in USA. I mispoke
I just had the same exact issue, unfortunately I was NOT running in docker.
I did find the same fixup-script as you in Digital Ocean, and hope I got rid of it.
I don't really get how they gained access to the server - that still puzzles me.
Anyway a slight question; I had the node red user account set up in a chroot-environment - my sshd_config has :
Match User nodeuser
ChrootDirectory /home/nodeuser
How is it still possitble for that exec to access stuff outside the jail ?
Also - is it possible to disable this node as an extra precaution ?
you can tell the runtime to ignore the exec node by adding the nodesExcludes property to your settings file:
nodesExcludes: [ "75-exec.js"],
I can't claim to have studied the script in any detail, but from what I saw of it, it didn't necessarily do anything outside the user environment: it downloaded a script to run and it configured that user's cron setup to redownload the script regularly.
But of course, none of that removes the need to properly secure your node-red instance with adminAuth so only you can deploy changes.
Personally I think that gives a incorrect “feeling” of security as the node-red instance is still insecure.
Whilst it inhibits the current attack vector, the fundamental issue is attaching an insecure machine to the internet. I’m sure other attack vectors would surface.
That is concerning. Someone found a way in and if you don't know how, there's a good chance the hole is still open.
I assume that you've gone through all the running apps to make sure there is nothing there? Also checked systemd and other startup script locations to ensure nothing has been slipped in?
Also, do you have SSH exposed to the Internet on the default port (22)? If so, I'd recommend moving that as it stops most of the automated bots trying to brute force a connection. I also recommend not exposing Node-RED on its default port.
Next thing to look at is what user ID's can log in remotely? Do you need them all? Do they all have strong passcodes (minimum 12+ characters, I'm using up to 24 now)? If you have any cert-based logins for SSH, I'd recommend generating new certs for them all since, if someone has/had access to your server, they probably also have your private keys.
And, of course, I'll assume that your system is fully patched? I tend to use Webmin (on a non-default port) and I make use of its auto-update features to keep everything patched.
Hopefully people already know that a new end-point on the Internet will start to get probed by attack bots within 30sec. SSH brute force attacks on port 22 and other common ports will start not long after.
