MQTT - TLS encryption example

In this previous post I used Certbot & Letsencrypt to secure a node-RED server, and wanted to use the same Letsencrypt certificates to secure MQTT communication with a remote server.
I also wanted to ensure that all local network devices could continue to communicate with the server without encryption.

To be clear, the mosquitto 'Broker' is to be installed & running on my Raspberry Pi, and the remote 'Client' is another node-RED instance running in a Oracle VM.

Secure MQTT normally uses port 8883, so using port-forwarding, open up port 8883 on your Raspberry Pi (broker). It is not necessary to open port 8883 on the client instance.

The Broker

Install mosquitto on the broker;
sudo apt install -y mosquitto mosquitto-clients

We must then tell mosquitto to listen to port 8883 and use TLS to handshake with the client. This is done by creating a mosquitto configuration file;
sudo nano /etc/mosquitto/conf.d/TLSconfig.conf and paste into it;

# Local MQTT
listener 1883

# Secure MQTT
listener 8883
## This is standard and should always be this
cafile   /etc/ssl/certs/DST_Root_CA_X3.pem
## These are from your installation of LE
certfile /home/pi/.node-red/certs/fullchain.pem
keyfile  /home/pi/.node-red/certs/privkey.pem
## Forces use of modern version of TLS to avoid security issues
tls_version tlsv1.2
## Force all clients in this listener to provide a valid certificate, change the node config to allow this from NR
#require_certificate true
## Stop all unauthorised connections
#allow_anonymous false
## Use password file
#password_file /etc/mosquitto/passwordfile

Save the file, then restart mosquitto - sudo systemctl restart mosquitto
Note the paths to the certificates. These can be changed if your certificates are at a different location, but use the full path to them.

The Client

Because the Mosquitto configuration (above) is not enforcing that clients must present a valid certificate, you will be able to connect to the broker without any certification whatsover, but by changing the Mosquitto broker configuration to;

## Force all clients in this listener to provide a valid certificate, change the node config to allow this from NR
require_certificate true

..and restarting Mosquitto, that rule will now be enforced, and any MQTT client must present a valid certificate before the broker will allow it to connect.
To prepare a node-RED MQTT node to use a SSL/TLS connection, the client must also have valid certificates of it's own, and add them to the MQTT node like this;

mq1

Then add a new tls-config;

mq2

That is all the configuration that you need to do in the MQTT node, apart from setting the topic, QoS & Retain options. It should now connect to your broker.

The default mosquitto MQTT listener is port 1883, which allows local network connection to be made without certification, yet because you haven't opened up port 1883 (port forwarding) in the brokers router, all external messages on port 1883 will not get through the firewall, only TLS encrypted 8883 traffic.

Enforce Security
By now, you should have a working TLS encrypted connection between Client & Broker, but to ensure that only your own authorised Clients are able to subscribe to your Broker, we first of all create a username/password pair, to authorise the client.
So back to the Broker - cd /etc/mosquitto and create a username/password pair by sudo mosquitto_passwd -c passwordfile username where 'username' is a name of your choosing. The command will ask you to also enter, and confirm, a password of your choice.
To ensure that it has been successfully created - cat passwordfile to view the username and hashed string password.

Now, we update the Mosquitto config file that we created earlier - sudo nano /etc/mosquitto/conf.d/TLSconfig.conf, and enforce the rules by un-commenting;

  1. require_certificate true - to only accept clients that use TLS
  2. allow_anonymous false - to disallow clients who do not have the username/password
  3. password_file /etc/mosquitto/passwordfile - provide a link to the password file

Restart Mosquitto to read in the new settings - sudo systemctl restart mosquitto and your Broker should now only allow subscribers who are TLS enabled and present the correct username/password.
To add those details to a client, simply add your username & password (the plain text password NOT the hashed string password) to the 'Security' tab in the node-RED MQTT node.

3 Likes

I don't think it needs to be opened on the client, unless you mean to allow outward traffic to 8883 on the client.

My assumption is that it needs to be opened to allow 'inbound' traffic to the client on port 8883.
Am I incorrect?

The inbound won't be to 1883. All connections are made from the client to server and the inbound reply will use a random port (or something, not sure exactly how that works). But you don't need to open a port for it as it comes under the ESTABLISHED rule (something like that anyway). Although it might appear that the mqtt broker sends data to the client it uses the connection that was setup by the client when it connects.
Try closing the port and see if it keeps working, pretty sure it will.

I thought that only applied to MQTT over websockets?

Consider the public mqtt brokers such as https://www.hivemq.com/public-mqtt-broker/. You can connect to that from your PC without opening any ports.

Thanks @Colin, you are quite correct & I've amended the example above just to open port 8883 on the broker. :+1:

1 Like

Nice. Thanks for the writeup Paul, most helpful.

Tried this before though without much enthusiasm and never got it working. Possibly to do with the chain cert. Never really wanted to expose my MQTT so I never tied it down.

One thing though, if your certs are from Let's Encrypt, you should be able to verify the certificate in the NR MQTT configuration. This is something that, in general, is a good thing to do because unverified certs are not terribly secure. Probably not really a security problem in your case but it is a good practice to get into. I wont go into the full reasoning behind this though I can explain to people if they want to PM me.

Indeed, even self-signed certs can be verified as long as you create a root certificate that you keep safe and generate a full chain from.

A nice addition to the Node-RED knowledgebase, thanks.

Yes, I was surprised too, but it doesn't verify for some reason. Could it be because only the CA certificate is added?

Actually, having thought about it, the TLS settings in the mqtt node appear to be for authenticating the CLIENT since it includes the private key.

I'm thinking that maybe you should turn that off and try to use the correct port.

Have you actually verified that the connection is on the secure channel?

Also, you might want to check out the following page which shows you that you possibly have your certs in the wrong order. The cert should be the chain.pem, it is called fullchain.pem on my installation. Then the cafile entry should contain the verified CA root from the OS which is, according to the mosquitto site: /etc/ssl/certs/DST_Root_CA_X3.pem.

OK, I can see that I'm going to have to bite the bullet and actually try this myself :sigh: good job the Mrs is out at her sewing class!! :rofl:

So, some experimentation and reading later. Here are some updates and simplifications.

For the Mosquitto configuration, you need to add/change /etc/mosquitto/conf.d/custom.conf (you can call the file anything. You also seem to have to add the default port as well if you want to retain that. Mosquitto uses default settings which include the standard port but they seem to be turned off once you add your own custom settings. Note the bits in angle brackets that you need to change:

# Default Listener: 1883
port 1883
# Bind the default listener to localhost only if you want to force external connections to be TLS only
#bind_address localhost

# Secure listener
listener 8883
# TLS
## This is standard and should always be this
cafile /etc/ssl/certs/DST_Root_CA_X3.pem
## These are from your installation of LE
certfile /<path-to-LE-cert-files>/fullchain.cer
keyfile /<path-to-LE-cert-files>/<private-key-name>.key
## Forces use of modern version of TLS to avoid security issues
tls_version tlsv1.2

## Forces ALL CLIENTs to provide a valid certificate - change the node config to allow this from NR
#require_certificate true

You then need to restart the Mosquitto broker with sudo systemctl restart mosquitto. You can check whether it has started the correct ports with sudo netstat -lptu | grep mosquitto which should give you 4 entries:

tcp        0      0 0.0.0.0:8883            0.0.0.0:*               LISTEN      17697/mosquitto
tcp        0      0 0.0.0.0:1883            0.0.0.0:*               LISTEN      17697/mosquitto
tcp6       0      0 [::]:8883               [::]:*                  LISTEN      17697/mosquitto
tcp6       0      0 [::]:1883               [::]:*                  LISTEN      17697/mosquitto

Note that you do not have to make any firewall changes on the Pi, the OS does that for you and will open both ports. You can check that from another Linux/Mac device (or Windows using WSL) with telnet <IP-NAME> 1883 and telnet <IP-NAME> 8883.


To connect securely from Node-RED, you need to configure the MQTT connection to use the TLS connection not the standard one. You also need to use the IP name rather than the IP address because otherwise, the certificate won't be valid.

image

Note that you need to set the URL and the port but you don't need to set the "Enable secure connection" flag. That lets you authenticate the Node-RED client connection to the broker (if you set the require_certificate to true for example).

Many thanks to Paul for giving me both the clues and the motivation to get this done. :smiley:


To monitor what is going on with Mosquitto, you can use the command sudo tail /var/log/mosquitto/mosquitto.log -f. This will show you connections and disconnections. If you need more information, you can change the log level in your mosquitto broker configuration file and restart the broker.

# Logging. Defaults to "error, warning, notice, information"
# debug, error, warning, notice, information, subscribe, unsubscribe, websockets, none, all
#log_type all
#log_type error
#log_type warning
#log_type notice
#log_type information
#log_type subscribe

once you guys have the "perfect" solution sorted... it would make a great addition to the cookbook :slight_smile:

1 Like

It looks pretty much the same as my example above except you've called your configuration file custom.conf, whereas I called mine default.conf (could call it anything so long as it has a conf suffix).

I'm surprised you don't need to set the "Enable secure connection" flag, I thought that was the whole point.

Are you able to now verify the certificate?

While similar, note the differences in the cert entries. The root cert verification is provided by Debian's trusted root cert list.

The other difference in the config is that I have forced TLS 1.2 to make sure you don't use an outdated, insecure TLS connection (e.g. TLS 1.0).

Yes. You could also split it so that different bits were in their own files if you like.

Nope, I think that the help info could do with some updating (hint to @dceejay :smiley:) to make that clearer. The "Enable secure connection" flag is for the client to authenticate to the broker rather than the other way around.

Actually, that is hard to say since I have no access to the internals of the mqtt nodes without a lot of faffing. So I can't prove that the client is validating the certificate chain right now. Maybe Dave knows?

Anyway, it works and now uses the certificates recommended by Mosquitto themselves.

:sigh: OK, I'll drop a note into the slack channel. I assume that I need to do a PR?

Yes please - (maybe start with the info update you just mentioned :slight_smile:

1 Like

PR submitted.

1 Like

Just updated the first post to include @TotallyInformation's comments;

# Local MQTT
listener 1883

# Secure MQTT
listener 8883
## This is standard and should always be this
cafile   /etc/ssl/certs/DST_Root_CA_X3.pem
## These are from your installation of LE
certfile /home/pi/.node-red/certs/fullchain.pem
keyfile  /home/pi/.node-red/certs/privkey.pem
## Forces use of modern version of TLS to avoid security issues
tls_version tlsv1.2
## Force all clients in this listener to provide a valid certificate, change the node config to allow this from NR
#require_certificate true

Just updated first post above to describe how to configure client node-RED MQTT nodes in order to present a valid certificate, and connect to the broker.

2 Likes

This is how I got MQTT SSL/TLS connections to a Mosquitto server using a domain certificate issued by Let's Encrypt working, including certificate validation in Node-RED:

Mosquitto configuration:

listener 8883
certfile /lets_encrypt_certs/domain.tld.crt
cafile /lets_encrypt_certs/domain.tld.chain.pem
keyfile /lets_encrypt_certs/domain.tld.key

The three files in /lets_encrypt_certs/ are created by a Let's Encrypt cert-bot.
Note: No require_certificate true - this is for authenticating the user in Mosquitto with a certificate.

Node-Red configuration:

  • Port: 8883
  • Enable secure (SSL/TLS) connection: Check.
  • CA Certificate: Provide a file that contains the chain for certificates issued by Let's Encrypt (see below).
  • Verify server certificate: Check (otherwise all this is pointless).
  • Server Name: The domain name for which the Let's Encrypt certificate was issued (this is crucial - otherwise any certificate issued by Let's Encrypt would be accepted).

Leave the rest unchanged/empty.

The certificate chain file for Let's Encrypt (save as text file and use for CA Certificate):

-----BEGIN CERTIFICATE-----
MIIFjTCCA3WgAwIBAgIRANOxciY0IzLc9AUoUSrsnGowDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTYxMDA2MTU0MzU1
WhcNMjExMDA2MTU0MzU1WjBKMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNTGV0J3Mg
RW5jcnlwdDEjMCEGA1UEAxMaTGV0J3MgRW5jcnlwdCBBdXRob3JpdHkgWDMwggEi
MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCc0wzwWuUuR7dyXTeDs2hjMOrX
NSYZJeG9vjXxcJIvt7hLQQWrqZ41CFjssSrEaIcLo+N15Obzp2JxunmBYB/XkZqf
89B4Z3HIaQ6Vkc/+5pnpYDxIzH7KTXcSJJ1HG1rrueweNwAcnKx7pwXqzkrrvUHl
Npi5y/1tPJZo3yMqQpAMhnRnyH+lmrhSYRQTP2XpgofL2/oOVvaGifOFP5eGr7Dc
Gu9rDZUWfcQroGWymQQ2dYBrrErzG5BJeC+ilk8qICUpBMZ0wNAxzY8xOJUWuqgz
uEPxsR/DMH+ieTETPS02+OP88jNquTkxxa/EjQ0dZBYzqvqEKbbUC8DYfcOTAgMB
AAGjggFnMIIBYzAOBgNVHQ8BAf8EBAMCAYYwEgYDVR0TAQH/BAgwBgEB/wIBADBU
BgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEEAYLfEwEBATAwMC4GCCsGAQUFBwIB
FiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2VuY3J5cHQub3JnMB0GA1UdDgQWBBSo
SmpjBH3duubRObemRWXv86jsoTAzBgNVHR8ELDAqMCigJqAkhiJodHRwOi8vY3Js
LnJvb3QteDEubGV0c2VuY3J5cHQub3JnMHIGCCsGAQUFBwEBBGYwZDAwBggrBgEF
BQcwAYYkaHR0cDovL29jc3Aucm9vdC14MS5sZXRzZW5jcnlwdC5vcmcvMDAGCCsG
AQUFBzAChiRodHRwOi8vY2VydC5yb290LXgxLmxldHNlbmNyeXB0Lm9yZy8wHwYD
VR0jBBgwFoAUebRZ5nu25eQBc4AIiMgaWPbpm24wDQYJKoZIhvcNAQELBQADggIB
ABnPdSA0LTqmRf/Q1eaM2jLonG4bQdEnqOJQ8nCqxOeTRrToEKtwT++36gTSlBGx
A/5dut82jJQ2jxN8RI8L9QFXrWi4xXnA2EqA10yjHiR6H9cj6MFiOnb5In1eWsRM
UM2v3e9tNsCAgBukPHAg1lQh07rvFKm/Bz9BCjaxorALINUfZ9DD64j2igLIxle2
DPxW8dI/F2loHMjXZjqG8RkqZUdoxtID5+90FgsGIfkMpqgRS05f4zPbCEHqCXl1
eO5HyELTgcVlLXXQDgAWnRzut1hFJeczY1tjQQno6f6s+nMydLN26WuU4s3UYvOu
OsUxRlJu7TSRHqDC3lSE5XggVkzdaPkuKGQbGpny+01/47hfXXNB7HntWNZ6N2Vw
p7G6OfY+YQrZwIaQmhrIqJZuigsrbe3W+gdn5ykE9+Ky0VgVUsfxo52mwFYs1JKY
2PGDuWx8M6DlS6qQkvHaRUo0FMd8TsSlbF0/v965qGFKhSDeQoMpYnwcmQilRh/0
ayLThlHLN81gSkJjVrPI0Y8xCVPB4twb1PFUd2fPM3sA1tJ83sZ5v8vgFv2yofKR
PB0t6JzUA81mSqM3kxl5e+IZwhYAyO0OTg3/fs8HqGTNKd9BqoUwSRBzp06JMg5b
rUCGwbCUDI0mxadJ3Bz4WxR6fyNpBK2yAinWEsikxqEt
-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----
MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4
WhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu
ZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY
MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc
h77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+
0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U
A5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW
T8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH
B5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC
B5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv
KBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn
OlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn
jh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw
qHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI
rU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV
HRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq
hkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL
ubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ
3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK
NFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5
ORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur
TkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC
jNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc
oyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq
4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA
mRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d
emyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc=
-----END CERTIFICATE-----

(These are the ISRG Root X1 (self-signed) certificate and the Let’s Encrypt Authority X3 (Signed by ISRG Root X1) certificate found at https://letsencrypt.org/de/certificates/.)

Screenshots:

grafik

grafik