What would be the best way to do this? I read that MQTT is not gonna be happy over VPN. My networking knowledge is leaving me at a loss as to which black hole to dive down on this one. I really just want to make a simple watch dog from my VPN server (lets call this flow 1) reading data from (flow 2) a remote 4g flow to throw a twilio alarm if I lose the tunnel connection (heartbeat timeout). I saw the post about remote/local connection but this does not seem to apply to my application. Which comms strategy would you guys recommend I pursue to accomplish this?
I haven't been able to actively try this yet myself (my one end of internet access is trapped behind an ISP firewall that they haven't figured out why my paid-for-service-to-allow-port-forwarding isn't working... anyhow...)
But @PeteKnight pointed out ZeroTier to me, it seemed simple enough to setup and he uses it with Node-red.
1 Like
Yep +2 for Zerotier for a Simple Point to Point connection. The beauty is that you can leave it running all the time and only the data destined for the endpoint IP (which is a private address) will go over the link
Gets a little trickier if you want to open up more nodes but still quite doable
Craig
1 Like
If you were going to use ZeroTier then you'd probably be able to replace your VPN with ZeroTier.
I use ZT like this...
I have a holiday home in Spain with a Pi running Node-Red 24/7.
I also have a mini PC that runs Windows and that's connected to a smart switch so that I can power it on when needed. (the PC is used for doing OTA updates to my ESP devices, and some other stuff that I cant do via the Pi).
Both the PI and the PC have ZT clients installed and as well as their local IP addresses they have ZT addresses on my ZT network.
In the UK I have a similar setup, a Pi and a PC, plus an additional Pi for development, all running ZT clients.
I also have ZT installed on a number of mobile devices (phones, tablets etc).
The free vision of ZT allows up to 50 devices to added to your network.
When I'm at home in the UK (or anywhere lese in the world for that matter) I can simply type [Spanish Pi's ZT IP Addresss:1880] into a bowser to access my NR editor for my instance of NR running on the Pi in Spain.
I can also add a connection to my Spanish Pi's Mosquitto broker from my UK NR instance, allowing me to integrate data from Spain into my UK flows. I've also set-up a ZT connection to my Spanish Mosquitto broker in MQTT Explorer, so I can easily view and publish MQTT data from my devices in Spain.
I used to use TeamViewer for remote controlling my PC's but it's extremely slow compared to an RDP connection over ZT, which is what I do now.
The only disadvantage to ZT is that if you run the ZT app on a mobile device it will disconnect from the network after a period of inactivity (presumably to save battery power). It's a simple task to go in to the app flick the switch to turn the connection back on, but its something to be aware of.
Pete.
2 Likes
What makes you think that MQTT wouldn't work over a VPN?
There are several types of VPN but a site-to-site VPN or always-on point-to-point VPN should both work fine. The point of such a VPN (as opposed to the ones that let you use a jump point in a different country) is to create a secure tunnel between 2 different local networks.
1 Like
This looks interesting but I do not want to add another service if its possible to do it over the VPN as I'm trying to put this in a production environment. I don't know how to expose my broker to the vpns subnet or how that architecture should work. @TotallyInformation From what I understand I need to create a broker that can be accessed from the vpn's subnet 10.8.0.x (openvpn). Why I'm confused is that this network doesn't exist unless the VPN is connected. Again not a networking guru. If I have my 4G network on 10.0.0.x with device at .175 and router at .1 my vpn subnet gives device 10.8.0.22. My VPN server is 104.131.x.x public ip and vpn ip 10.8.0.1 where do I put my broker? I don't know which subnet to put it on or which side to put it on. I think I can run it on my server or my client and subscribe from either end but I may not be understanding this correctly as Ive never tried to use MQTT before let alone across a VPN. edit doing my MQTT homework now
Thanks for the information pete, the disconnecting from the network after inactivity is a deal breaker.
So you’re running one of your Node-Red flows on a phone or tablet?
Pete.
User will be running the dashboard on a tablet like an HMI. I'm trying to emulate an HMI so the user should not have to interact with the screen at all to keep it online or updating. The trend screen will be looked at (at a glance) -at will- this is a minor inconvenience to us but will be unacceptable in customers eyes as there will be other devices they are also paying attention to. For reference, my dive into this is to replace a very expensive Allen Bradley system that is total garbage IMO. I've had a working proto type for a few years on a beaglebone and now I've moved to an Opto22 Groov Rio for the 'big leagues'. I'm hesistant to use any peripheral un-proven technologies. OpenVPN and Node Red have been accepted by the industrial world as of late. My industry is far behind and playing catch up 
If I can do this through the VPN then that is what I would prefer as Julian suggested.
But your question was about MQTT over VPN, and your dashboard will presumably be delivered via HTTP(S) over your VPN?
Maybe a more detailed description of your setup, hardware and objectives would have been useful.
Pete.
Pete,
Sure, I apologize for lack of information and any 'harsh tone' I may have, I appreciate the suggestions.
I have a modbus device on a 4G router being read by an Opto22 Groov Rio. I'm using OpenVPN and a Digital Ocean Ubuntu v20 server to dial out my tunnel. The objective is to allow remote monitoring to the device via tablet (or phone) as this is a field device that is being moved around a plant. I'm using Twilio to send out alarms from the RIO according to given conditions. This all works fine and great. What we do for comms alarms in PLC's is we send bits at a given interval from one PLC to another and if that bit doesn't come through a timer times out and we throw an alarm. This can cascade various actions like shutting off outputs etc etc. What I'm trying to do is throw a twilio alarm from my VPN server if my remote device loses its connection. Yes, I'm delivering HTTPS over my vpn but I'm struggling to figure out a good way to send data between my flows so that my server knows my remote unit isn't connected and can alert an operator. I think I should be able to host a broker on one side of the tunnel and subscribe to it on both sides and push a heartbeat. That is the theory at least. This is really easy to do with a plc but we are always on hardwired lans. So far...I've created a Mosquitto broker on my server side and tried to connect to it through the tunnel but thats not working which is why I was asking about routing and subnets. I'm currently reading documentation about password auth and plugins etc etc. just wanted to know if its possible before I waste a bunch of time and if I'm thinking about it correctly.
Unfortunately, with this kind of a setup, being a network novice is probably not going to work for you. You will need to learn at least a little.
Firstly, you need to understand the layout of your networks. As I see it you have 3. The central server is on a VPS and so accessed over the internet. The IoT devices are on another network and then there is the user tablet, that is the 3rd. Of course, your IoT devices might be many and all on different networks.
A VPN is designed to create a bridge between local networks (LANs) over a wide area network (WAN). But to do that, your LANs have to be compatible. Firstly they either must share an address space or you must create routers that will link things together.
Your VPS is likely connected fairly directly to the Internet (as far as you can see anyway). That gives you an Internet address and a 127.x.x.x localhost address space. When you set up your VPN endpoint, you probably give it a non-routable address either 192.168.x.x or 10.x.x.x. Lets say 10.0.0.1 for the sake of argument. This also has the MQTT broker on it.
So on the network that your IoT device(s) are on, might be on a non-routable address space as well. Lets say 192.168.1.10 as our example.
Somewhere on that IoT network, you also need to run a VPN endpoint but you might now have 10.0.0.2 which cannot be connected to from your IoT devices. In that case, you must create a router that routes from 192.168.1.x to 10.0.0.x. Now, when you set up your IoT devices, you need to give them the address of the broker but it is on the wrong network. So now you need to create a link. Your router needs an address, lets say 192.168.1.200 with 1883 (the port the broker uses) that routes to 10.0.0.1 port 1883. You configure your IoT devices with 192.168.1.200 as the broker address and the networking sorts the rest out.
On the tablet, things are much easier because your VPN client will put the tablet straight onto your 10.0.0.x network. This is the reason I'm always telling people to be cautious about VPN's - with tablets, laptops, etc, your VPN client is connecting that vulnerable, mobile device direct into a supposedly trusted network - this is generally "Not a good thing"(tm). Unless you have taken great care about the security of the remote devices.
Hackers love device-endpoint VPN's - well the type of hackers that make the REAL money or, as likely, work for a government. So you can see that the risk from a device endpoint VPN connection is lower if you are not a "target of interest". Maybe even low enough that you won't worry too much about it. Still worth remembering though and not assuming that a VPN is a security feature - it all too often is not.
Anyway, hopefully I got all of that right. It is Friday evening after a very long and trying week. So I am tired and I may have got some things wrong. This is not professional advice - check everything out before assuming I might be right 
1 Like
Julian, I would say I'm a novice + Thank you for the explanation. I do have end points of 10.8.0.1 (VPS) and 10.8.0.22 (iot device). The LAN ip for the iot side is 10.0.0.x. So you are saying I need to route my router (10.0.0.1:1883) to my device vpn end point (10.8.0.22:1880) and then set my 'broker address' in my MQTT node to my router address (10.0.0.1). I can try port forwarding on my router. I'm not sure if I should set my mosquitto broker to my tunnel as well 'listener 1883 tunnnel ip' or just listen on 1883. Trying different stuff. Thanks for the input and Have a great weekend!
As Julian has said you need to be very careful here as the remote VPN device (Tablet) will have full access to your network.
You do really need to contract this out to someone who know what they are doing but in a nutshell
-
Create a VPN between your tablet and your Broker (you mention Digital Ocean) and setup firewall rules to only allow 1883 (MQTT) across the VPN - so the tablet can only talk MQTT into VPN and to the broker
-
Setup a 2nd VPN between your end devices that produce alarms and alerts etc and only allow MQTT across this link - i.e the devices producing information can push it to the Broker IP address only
This way if a tablet does get lost and become compromised it can only attack the broker and no other devices.
Personally when i do this i actually have two brokers and a client in a DMZ that can only talk to the brokers. It takes information from the "internal" broker that the devices talk to and puts it across to the broker that the exposed devices communicate with.
Craig
2 Likes
No, your IoT network is already in the right network. I did leave out something important while trying to keep things simple. Something called a netmask.
Most small to medium networks will use a netmask of 255.255.255.0 which allows any device on your 10.8.0.x network to reach any other. Any device with an IP address from 10.8.0.1 to 10.8.0.254 should be able to talk to each other.
So assuming your VPN is up, you should be able to SSH into your VPS and run ping 10.8.0.22 and you should get a response.
By the way, your router may not help you anyway - it depends on how your VPN is set up.
Without knowing the exact configuration of everything, it is hard to give an exact answer. But if everything is on the same network, it should all work. If it isn't working then it is likely being blocked by something we don't know about.
Networking can be quite complex as you can see. Secure networking is more complex than most people allow for (including many networking "experts").
1 Like
"As Julian has said you need to be very careful here as the remote VPN device (Tablet) will have full access to your network." I understand this.
"You do really need to contract this out to someone who know what they are doing" thats no fun
@craigcurtin Regarding suggestion 1 now we are talking about the fun stuff. Why would I only want a dedicated VPN for MQTT port? This is interesting from a security perspective. I'm basically only grabbing modbus 502 reads on the LAN side of the iot device. I'm curious what the logic is for having 2 VPNS...reducing attack vectors? You say as much but this seems excessive (i'm prob wrong :D)
If the tablet became lost I would just remove the user from the VPN user list, but I assume you are saying that in that time window that it becomes lost and I discover that it is lost: do not allow an attack vector.
Additionally, since MQTT is a new protocol to me I'm curious what the problem is if the broker gets attacked at all. Even if it gets comprimised they can't do anything to the device/machine. Is the worry message spoofing bad messages? From my perspective I almost don't even care if they wreck the broker as long as they can't modify the device or write to it then its a wash but I'am curious what others are thinking in their applications. Don't get me wrong it would be bad if users were getting bad messages and data but it wouldnt be the end of the world. The real bad stuff is if an attacker started writing to modbus addresses. The device I'm reading will do its thing regardless of what messages I get back on mqtt in other words. Its more just lost time.
@TotallyInformation I know what netmasks are and CIDR I just rarely have to touch networking stuff this complicated. I can absolutely ssh into my VPS and ping my device endpoint @ 10.8.0.22. I'm using a TUN (routed) VPN. I have setup a TAP (bridged) vpn before but it was a pita and ended up not even needing it for what I was doing.
Looking through mosquitto docs it seemed I could have the broker listen on the tunnel by specifying listener port ip address. This was not working. "Networking can be quite complex as you can see." Yes 
I was going to try to get it talking 'open' before I do any securing of the network which is usually what I do. I don't know if I can subscribe to the broker with a routed vpn but I'd think I can if I setup my routing correctly which is why I posted here in the first place to see If I'm barking up the wrong tree. I admit I don't understand where on the network layer the mosquitto broker is sitting once I start it.
My end points are as described 10.8.0.1 (VPS) and 10.8.0.22 (Device). Device lan is 10.0.0.x with router at 10.0.0.1. Device is at 10.0.0.175. My vpn is over 443 and using another server for the CA. What Julian was saying made sense to me that I need to push a route between the end point to the lan on 1883. I'm not in the office this weekend it may be being blocked by UFW on the RIO which I can check on monday.
Appreciate you gentlemen getting the wheels turning and making me rethink network architecture.
There are some important assumptions buried in that text. And some not uncommon ones.
-
Assumption that access to MQTT broker could do no real harm. Really? Are you sure? If not now, what about in a year or 2 when the service is more mature? What is being controlled and what intelligence is being sent back to the centre? But also, we don't really know what type of service you are offering. For all we know, you might be taking smart-building information from a government building? And even if you are only receiving sensor data now, what about later when you decide to use this service for controlling a building or some critical piece of industrial equipment?
Of course, if the data isn't important. If attackers couldn't glean something useful or combine with other data to get something useful. If your customers aren't important. Then yes, sure, the risk is low.
-
Assumption that it is only the broker at risk. Nope! Every weakness is an entrypoint for attackers. They like nothing more than discovering a weakness that lets them get a foothold and then traverse slowly but surely to more important systems or data.
You are creating a very flat network so make certain that it has very limited reach. A "good" network design would isolate each part of the network. Each building's IoT devices on 1, end user devices on another, servers on yet another. And, as previously mentioned, you might go further and have semi-trusted servers in a DMZ with fully trusted servers reaching out and getting data from them (but never allowing anything to reach into your trusted network). And between each network, a firewall. Or, if the value is low, at least have them on separate VLANs with a router/firewall to route and restrict traffic flow.
Again, is this a high risk? Maybe not. Again, depends on the circumstances. The value of the data, the value of what the data represents, the value of the intelligence you are putting together, the value of your server (do you want bad people using your server for nefarious acts?), the value of your customers and the value of your reputation.
So will you be attacked if you leave a door open? Almost certainly. Attacks start with automated discovery. Will that result in data loss or other bad things? Probably not (with some notable exceptions - a platform on which to run stuff is VERY valuable to people doing bad things over the Internet).
Ouch! Unless you can get that first part done in less than 30 seconds or you are doing it on a closed network away from the Internet, that is not a good approach. 30s is how long it takes for a server on the Internet to be discovered by the auto-bots looking for entry. Maybe less now.
My non-professional advice for what it is worth, do your experimentation on a closed network. And if what you are doing is worth real money or reputation, get someone to security test it.
And of course, a final disclaimer. I am an Enterprise Architect not a networking expert. I do and have done a lot of security as well but I absolutely know that I know enough to know that I don't know enough!
3 Likes
OK Julian has covered most of it but here we go
(No offense) but you are obviously not a security professional so if you roll out something today that seems pretty secure - you are probably not going to keep up to date with all the latest vulnerabilites and exploits that come out and patch accordingly in a timely fashion.
You therefore need to architect it in the most secure fashion possible.
Whilst one of the concerns is someone losing a tablet/device with the VPN embedded - a bigger concern is one of those devices being compromised and you not knowing about it. You therefore need to limit from the end you control exactly what can come down the link.
By putting a seperate Broker out for these devices to interact with and only allowing data to be pushed to that broker from internally then if (and when) it does get compromised it will not be able to be used as a platform to launch an attack on you/your clients networks.
If on their internal network you then have a broker that the IOT devices connect to and upload their data to you can then choose what data is pushed from there to the external facing broker
I would usually (depending on the size and what they do) actually have a device on a seperate VLAN with a 2nd interface on the internal broker that would be used to get data from there and post it to the external broker rather than allowing the Internal broker itself to push data to the external broker on a seperate VPN
Craig
1 Like
"For all we know, you might be taking smart-building information from a government building?" - True, I'm not right now but would want to take this further in the future. One thing to note though that this is not connected to any other devices/networks on either end currently.
When I mean get it talking 'open' I think we already agree. My methodology isn't as slapdash as it may sound. I will perhaps open a port on the fire wall try something and if it doesn't work, close that port. This round trip may take 30s or a few minutes. I'm not operating without any firewall and without fail2ban. I'm aware of potential risks associated with opening and leaving random ports open.
"I am an Enterprise Architect not a networking expert." That is very cool, but I assume this job requires you to come up with network topologies that 'networking experts' then implement? So you will be somewhat familiar with security issues associated with various protocols. -Which is one reason I posted this in the first place- Im more doing R&D. I had no idea I'd be opening such a can of works delving into MQTT but such is life. I would say if I showed you some of the network topologies and security strategies I have seen that are -in place- you would be appalled. I'm trying to do my due diligence and not add to the problem.
"(No offense) but you are obviously not a security professional so if you roll out something today that seems pretty secure" - none taken and you are correct sir.
"I would usually (depending on the size and what they do) actually have a device on a seperate VLAN with a 2nd interface on the internal broker that would be used to get data from there and post it to the external broker rather than allowing the Internal broker itself to push data to the external broker on a seperate VPN" Looking for clarification in this statement. You are saying 2 brokers, 1 internal on lets say VLAN 2 that has a route to the device that is on VLAN 1. "used to get data from there and post it to the external broker rather than allowing the Internal broker itself to push data" from what I'm reading you bridge brokers together, not understanding the difference in your langauge here between 'posting to external broker' and 'pushing data'. From what I'm understanding about connecting mosquitto brokers I would only have a bridging mode as a connection type. Trying to understand what you mean.
Additionally based on your advice I have reached out to a networking consultant to advise me on this. I tend to try to do as much as I can and learn as much as I can with new technology so I'm not in the dark on these things. Even understanding it in theory goes along way and I learn by doing, as I'm sure many people do on this forum. Again appreciate the advice.
OK, I need to make sure people understand is all, too many people don't.
Absolutely, all I am saying is that what I might suggest here is just that, a set of ideas. It isn't a design nor could it be considered secure without specialist testing. e.g. what I say in this forum is hopefully reasonably accurate but it isn't professional advice. When you wire up that government building and if it goes wrong, don't come a-calling 
To be fair, this isn't an MQTT issue in any way. This is a networking security issue. It would be the same if you used some other protocol.
But not surprised. I had a networking and an infrastructure specialist come up with a datacentre design the other day which was shocking as well. They certainly should have known better. I also come across far too many "secure data centres" that turn out to be cupboards that the cleaners have access to and that have no protected power nor filtered aircon.
And good on you. Please don't take our comments as a personal attack, they certainly are not and we are happy that you are looking for help. But we are just pointing out that this is an Internet discussion forum and not a professional support helpdesk 
I know now that you understand that but we have to be clear sometimes.
A great approach. One I fully endorse and use myself very regularly I doubt that you would be surprised that I often surprise our specialist vendors 
If ever we meet, I would no-doubt regail you with some of the stand-up arguments I've had with supposed professionals who have had to back down when I presented them with the evidence. Some of whom even went on to become good friends. Indeed, I had a very "robust" discussion with my boss and some senior tech and account people from Apple on Friday - they can be very aggressive about their product approaches and get very defensive when taken to task about standards that the 80% of users have access too but Apple users do not (current worldwide Apple market share on the desktop being around 8%).
I wish you all the best on your endeavours. And now we know where you are up to and what you understand, we may be able to help further.
1 Like
