Nodered hacked by adding invisible nodes

[kevinGodell]

@iznogoud320

I don't know and I haven't kept a copy of nodered settings to check .
But yes i did use nodered tool to hash it.

I would suggest that you still password protect the editor even if you have hidden your node-red from the outside world. Your cat may attempt to hack you from within your network.

If you want to test the hashing time duration on your machine when using different costs, you can play with these flows. Warning, the function node will require the installation of bcrypt which may or may not already be installed on your system.

[{"id":"02ad12a4dd3bb656","type":"function","z":"bf43e3aa151c3e51","name":"bcrypt hash","func":"const { hash } = bcrypt;\n\nconst start = Date.now();\n\nmsg.hash = await hash(msg.password, msg.cost);\n\nmsg.duration = Date.now() - start;\n\nreturn msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[{"var":"bcrypt","module":"bcrypt"}],"x":370,"y":220,"wires":[["8d464f555326b98d"]]},{"id":"47c177b7330ab163","type":"inject","z":"bf43e3aa151c3e51","name":"","props":[{"p":"password","v":"5up3r 53cr3t pa55w0rd","vt":"str"},{"p":"cost","v":"16","vt":"num"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":170,"y":220,"wires":[["02ad12a4dd3bb656"]]},{"id":"8d464f555326b98d","type":"debug","z":"bf43e3aa151c3e51","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":570,"y":220,"wires":[]},{"id":"4c1503c8e7b7592f","type":"inject","z":"bf43e3aa151c3e51","name":"","props":[{"p":"password","v":"5up3r 53cr3t pa55w0rd","vt":"str"},{"p":"hash","v":"$2b$16$4nWRWwHiEufnUV.o3K7e0uwhT4x9G7wKsfXL9RsDZvXrLqsYFFJQG","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":170,"y":300,"wires":[["7fe9f0d5f22f37a4"]]},{"id":"7fe9f0d5f22f37a4","type":"function","z":"bf43e3aa151c3e51","name":"bcrypt compare","func":"const { compare } = bcrypt;\n\nconst start = Date.now();\n\nmsg.compare = await compare(msg.password, msg.hash);\n\nmsg.duration = Date.now() - start;\n\nreturn msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[{"var":"bcrypt","module":"bcrypt"}],"x":380,"y":300,"wires":[["bb78be24f86cbd23"]]},{"id":"bb78be24f86cbd23","type":"debug","z":"bf43e3aa151c3e51","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":570,"y":300,"wires":[]}]

results from my pi4:

password: "5up3r 53cr3t pa55w0rd"
cost: 16
hash: "$2b$16$jelKXI4R3WsJtCGWThEk4OUO2brGAmZXeMUlCEgY.cMrA0oZ6v/66"
duration: 8253

And then later when using this password, node-red took about 8 seconds to authenticate me and start the session. The only delay is on the initial login. That would make any brute force attempt to take much longer. Further testing showed me that the default cost of 8 only takes about 30 milliseconds to hash the password, versus 8000 milliseconds.

[jodelkoenig]

@kevinGodell

I would suggest that you still password protect the editor even if you have hidden your node-red from the outside world. Your cat may attempt to hack you from within your network.

I was wondering whether the rest of your post doesn't apply to me since I neither have cats or dogs in my house ... mh

On a more serious side: Thanks for your write up, Kevin! Cool!

This small example clearly shows again how little I know about security. And that's exactly why I use VPN/zerotier to gain remote access to my dashboard/editor. Latter beeing secured as descibed in nodered's documents. So I will definitely give it a try.

[Paul-Reed]

@kevinGodell

That would make any brute force attempt to take much longer

Would something like fail2ban be a better option, which would block brute force attempts.
So users could use a relatively low cost auth, as hackers would only have a limited number of attempts before being blocked, and therefore unlikely to succeed.

[jbudd]

I have watched hack attempts on a linux machine to which I had opened port 22 (ssh).

They tended to come in waves, just a few login attempts using different account names and switching IP addresses, with a few days before the next wave of attempts.

Clearly this behaviour is designed to evade protections such as fail2ban.
Maybe they switched to this stealthier approach after noticing that fail2ban had blocked them.

I think @kevinGodell's suggestion of a password hash which takes 10 seconds or so to verify is likely to be at least as effective at keeping attackers at bay.
I'm not clear how often NR prompts for the password though. If you login today, will access from the internet tomorrow be unchallenged?

Also, in my experience, using a port number way out of the normal range such as 52997 is effective since scanning for open ports takes time.
Not having a username called "root", "pi", "admin" or "user" is a good idea too (not relevant to hacking Node-red).

Of course, none of these are as effective as not opening any ports in the first place.

[TotallyInformation]

@jbudd

I have watched hack attempts on a linux machine to which I had opened port 22 (ssh).

If you really must connect Node-RED to the outside world or SSH for that matter - PLEASE make sure you change the default port to something less obvious. In the days when I ran various VPS's, the first thing I did was to change SSH away from port 22. This meant that I never got the constant bashing on that port by the many bots looking for entry.

Since Node-RED is now also obviously a target, at least change to a different port, preferably some odd high number.

@jbudd

Not having a username called "root", "pi", "admin" or "user"

Yes, any device connected to the Internet should not be using those. Certainly not pi and though root is hard to avoid, at least ensure that it cannot be accessed via any external entry point and that it has a very strong passcode. Or better still only allows certificate logins.

With the variety of free tier 3rd-party cloud VPN type solutions available now, there really is NO EXCUSE for exposing anything valuable without going through them.

[iznogoud320]

I'd like to thank everyone for all the advice.
For the moment here is the solution I implemented:

I have 2 router, Router 1 being my ISP router and router 2 dedicated to iOT on which is running my nodered server.

• nodered editor long passsword protected and nodered default port changed for a big number.
• nodered port routed on router 2.
• on router 1 VPN wireguard activated and fixed IP specified.
• on nodered server (Pi), firewall configured to only accept connection from local network or the VPN fixed IP.

Therefore nodered should not be exposed to the internet except from VPN connection.
Hope it will be fine this time.
Thanks again to all

[GogoVega]

@iznogoud320

nodered port routed on router 2.

I'm not an expert and I may have misunderstood

I've always heard that a router with an open port is a risk, having two routers doesn't change anything because you have port forwarding - it's like the first router opens the door.

The best VPN I know is the router itself, you establish a connection to the router and the router manages the ports

This is just an opinion and Julian is more knowledgeable than me in the field.

[iznogoud320]

As the VPN server is on router 1 I can access local network of router 1 (ISP router). I use the public address of router 2 and port to access the nodered server located on local network of router 2.
I cannot add routing on router 1 to router 2, because it's my ISP router and it doesn't have this functionality. But maybe there is another solution.

[meeki007]

Security yourself is hard. There is not easy solution. If any one reading this post would like to dedicate themselves to learning one way (how I currently run node-red) of locking down node-red so you can run the editor on the nasty web here is a path.

1. Install the node-red instance in a docker or VM - if node-red is needed on hardware like a pi to flip switches then write a API to communicate or use mqtt to and from it to the docker or VM instance hosting the editor to the big bad internet. Now if the instance gets hacked its not an issue if you not passing no no commands back to the API.
   Password length for node-red should be over 28 char with all sorts of symbols, upprer and lower case .... etc. Ummm I use over this number length by a good amount.

2. Install nginx use ssl_dhparam and ssl_protocolsTLSv1.3; There is allot on your own time you need to read into: limiting amount of connections and rate limiting ...etc. But most important is your going to reverse proxy to your node-red instance exposed to the big bad internet.
   When you do this change the httpAdminRoot: in node-red settings file to some new location. This keeps people from hitting your editor out of the gate.
   Keep you dang editor location from ending up in a search result!!! in nginx I use.
   location = /robots.txt {
   add_header Content-Type text/plain;
   return 200 "User-agent: *\nDisallow: /\n";

3. Use fail2ban to detect 401 failures from nginx this way if a bad actor is trying to brute force password attack your node-red editor instance it will automatically ban them. <---For me this is always going off. People from Russia and China are hitting this trap daily!
   Use fail2ban to stop port scans!!! I created a rule so after 10 failed ports they get banned.

4. lock down the editor and flow file when not making changes!
   in the settings.js file set disableEditor: true when your not actually changing the editor/flow file.
   I also lockdown the flow.file making it immutable in linux.
   $ sudo chattr +i $HOME/.node-red/flow.json
   This way you cant update it. Just unlock it when you plan on using the editor.

There is allot more that can be done but this is a good basic outline of the minimum I would do before running node-red on the big bad internet.

If you have critical service that you cant ever suffer from attack you can do what i do and setup wireguard and do not ssh into your server. Oh and if you do use ssh change the port and use key's and disable password auth.

Prepare to learn ............ or just pay a service company to do the work for you.

[gregorius]

@iznogoud320

Yes nodes were exec nodes and function nodes. They were all declared as hidden and positionned at coordinates 9999,9999.

Would a search using type:exec then actually discover these nodes?

How can I check my flows for potential hackedification?

[iznogoud320]

What I did is download a json export of the flows and searched the file for '9999', 'hidden', or the begining of the IP address.
Search on "Type:exec" worked but there were some function nodes too.

[gregorius]

Thanks for the tips

@iznogoud320

"Type:exec" worked but there were some function nodes too.

I rarely use exec nodes so for me that is a first step - I assume that I can't use the builtin search to search for x:9999 ...

If I created a flow for checking for this ... would that be the first virus scanner for Node-RED? Might there be a market for virus scanner nodes?

[gregorius]

Here's a simple flow for checking for nodes that are off-screen or exec type

[{"id":"bb658730abc8da93","type":"GetFlows","z":"2c6cdb05d5945f04","name":"","flowVersion":"v1","useAuthentication":false,"apiUsername":"","apiUsernameType":"env","apiPassword":"","apiPasswordType":"env","x":708,"y":628,"wires":[["94bb3080776d75bc"]]},{"id":"281f97b2bcfdb92b","type":"inject","z":"2c6cdb05d5945f04","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":503,"y":699,"wires":[["bb658730abc8da93"]]},{"id":"94bb3080776d75bc","type":"json","z":"2c6cdb05d5945f04","name":"","property":"payload","action":"","pretty":false,"x":919,"y":524,"wires":[["877ca7dc78a3dbdb"]]},{"id":"877ca7dc78a3dbdb","type":"function","z":"2c6cdb05d5945f04","name":"off-screen nodes and exec nodes","func":"msg.payload.forEach( nde => {\n\n    if ( nde.x > 8000 || nde.y > 8000 ) {\n        node.send({ ...msg, payload: nde})\n    }\n\n    if ( nde.type == \"exec\") {\n        node.send({ ...msg, payload: nde})\n    }\n\n})\n\n","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":1162,"y":753,"wires":[["f99be78bc5189649"]]},{"id":"f99be78bc5189649","type":"debug","z":"2c6cdb05d5945f04","name":"debug 66","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1316,"y":398,"wires":[]}]

it's based on the GetFlows from the introspection package and basically contains a function node that does this:

msg.payload.forEach( nde => {

    if ( nde.x > 8000 || nde.y > 8000 ) {
        node.send({ ...msg, payload: nde})
    }

    if ( nde.type == "exec") {
        node.send({ ...msg, payload: nde})
    }

})

very basic

[TotallyInformation]

@meeki007

Use fail2ban to detect 401 failures from nginx this way if a bad actor is trying to brute force password attack your node-red editor instance it will automatically ban them. <---For me this is always going off. People from Russia and China are hitting this trap daily!
Use fail2ban to stop port scans!!! I created a rule so after 10 failed ports they get banned.

Be great if you could document that setup somewhere so that others can do the same?

@iznogoud320

searched the file for '9999', 'hidden'

This would be nice in an independent script on a CRON, you could check every day (or more often if you wanted of course).

[meeki007]

Been documented for a while. This howto is from years ago. Times change but the fail to ban section and the 400 banning section is still good.

meeki007/node-red-server-howto-guide/blob/master/README.md#fail2ban_403_protection

[rko]

Would be nice to have some built-in brute force detection. This attack probably could have been prevented. Shouldn't be too hard to implement, no?

Yet again, the mentioned hardening methods are not for beginners.

[TotallyInformation]

@rko

Would be nice to have some built-in brute force detection

Do you mean in Node-RED or in @meeki007's write-up?

[meeki007]

@TotallyInformation

Do you mean in Node-RED or in @meeki007's write-up?

For brute force in the current how-to I linked above it only covers password spamming.

For more protection I use port scanning block to keep them out of the back-end of the server. This is not node-red specific but it helps them from finding all your services available on ports.

This is not well documented any where I've found on the internet. Mainly dealing with ports you need open.

Assuming you have fail2ban installed and your using UFW for your firewall:
ssh port is on 52009
http port is on 80
https port is on 443

Change the port for monitoring [sshd] rule section add a new rule under it for blocking port scans.
$ sudo nano /etc/fail2ban/jail.local

NOTE: [sshd] section add 
enabled = true
change port to 52009

NOTE: adding port scan blocking, add this under [sshd]
[ufw-port-scan]

enabled   = true
port      = 0:79, 81:442, 444:52008, 52010:65535
filter    = ufw-port-scan
banaction = ufw
logpath   = /var/log/ufw.log
maxretry  = 20

Notice how in the port ranges I've made sure to not include ports I use or want people to get banned for using.

Create new filter
$ sudo nano /etc/fail2ban/filter.d/ufw-port-scan.conf

[Definition]
failregex = .*\[UFW BLOCK\] IN=.* SRC=<HOST>
ignoreregex = 

Then Restart fail2ban
$ sudo service fail2ban restart

After the service restarts check to see if your new rule is working
$ sudo fail2ban-client status

I'm trying to think of all the tricks I'm using that I'm willing to share. Ask and I will provide. Maybe someone will take all this knowledge and write up a supper secure howto.

Status for the jail: ufw-port-scan
|- Filter
|  |- Currently failed:	4031
|  |- Total failed:	40900
|  `- File list:	/var/log/ufw.log
`- Actions
   |- Currently banned:	0
   |- Total banned:	6
   `- Banned IP list:	

This is a current ban for port scanning on one of my servers. Its funny over the months the number keeps going down. For a while there I had 100+ bans.
I think the bots have scripts to stop banging on hardened systems and move on to lower hanging fruit.
I also think the groups share their own black list for servers ip's not to waste time on because multiple IP's all at the same time stop scanning at the same time on a bot network.
My log file shows scans mostly happen in the less than 9K range. also spcifc targeted ports. So If I was going to put my ssh port out there it would mostly fall in the 20K to 50K range to just be safe.

[meeki007]

@rko

Yet again, the mentioned hardening methods are not for beginners.

I feel you on that!

Issues with making security baked into node-red:

1. as node-red is open source any measure created to stop something is just a well documented attack vector to be broken. By implementing external measures they can be different by each creator. Security by obscurity is not the best method but when combined with known working practices it makes each person's server a new nut to crack. Most hacks are just script kiddies bots banging on doors looking for the low hanging fruit.

2. breaking changes. If node red starts baking in security it causes server admins to loose their shit. Having to bypass or figure out how to work around changes can be a pain. I have node-red instances out there still running ver.0.18.x because of this. I keep them dockered or use custom API's to talk to them and never let them see the big bad internet.

3. Development time. Security is hard and very time consuming to code up. It takes a multiple more ammount of time to write security related code than it does to write a new node feature .... well most of the time.

4. Lastly, entry barrier for new users. If you move the security to node-red It will require new users to learn allot more to get node-red up and running. Reading allot more documentation. And issues when creating their setup. Allot of the issues/learning your having with securing node-red using outside resources will just be pushed into node-red.

I stated this many times and its not directed to you @rko ... its for all. With internet security now adays you have 2 choices.
LEARN LEARN LEARN for months ... and then keep leaning because new vectors will happen.
or pay someone else to do it for you -----> https://flowfuse.com/

[rko]

Well it looks like the feature I was referring to is already implemented, sorta

Just need to locate the section of code where the 10-minute duration is hard-coded and set it to a very high value in order to disable the account.

node-red/node-red/blob/0e8d3127944a91f39fb49bcb4b388b184ce11edd/test/unit/@node-red/editor-api/lib/auth/strategies_spec.js#L331