Node-RED SSL reverse proxy w/ Google Oauth

Share Your Projects
1

edit: please see this post for most recent deployment instructions

Here's a docker-compose.yml file to deploy Node-RED and Traefik, a reverse proxy that automates fetching, issuing, and renewing free SSL certificates from Let's Encrypt.

There are two ways Lets Encrypt verifies that you are the owner of a domain so they can give you a free cert.

  1. HTTP validation. Let's Encrypt gives you a token, and you put a file on your web server at http://<YOUR_DOMAIN>/.well-known/acme-challenge/token.... Lets Encrypt comes and checks that the token is there. Therefore, Let's Encrypt needs access to your server on port 80, which means you have to leave that port open and forwarded to your docker host.

  2. DNS validation. Let's Encrypt doesn't need to access your IP at all. You run a piece of software that connects to your DNS host, and enters a temporary TXT record with a token. Lets Encrypt hits the DNS and looks for the record. After 5 mins the record expires.

By far, DNS validation is the best way, and the only way I recommend. Only use HTTP validation if for some reason you can't change your DNS registrar.

I have some domains bought on google domains, some bought on namecheap, some bought all over. Namecheap, for example, has an API that traefik can use. But it really doesn't matter what DNS provider you use... log into it and forward it to Cloudflare. Cloudflare works with Traefik, every time. So go there and make an account and enter in your domain and follow the instructions you see. For now leave it on "DNS ONLY".

So, here are the requirements:

  1. You own a domain that you have either pointed to cloudflare for the dns method, or otherwise pointed to your public IP.
  2. If using cloudflare, you have a wildcard 'A' record pointing to your public IP, and have a zone API key and main API key
  3. If HTTP validation, port 80 and 443 are forwarded to your docker host
  4. You have a docker host ready and have deployed swarm mode (you only need one host)
  5. You have created a docker overlay network named 'traefik-public'

once those requirements are met, ssh into your docker host, and create a file called nodered.yml with the below contents (fill in all the variables)

[user@docker1 ~]$ docker stack deploy -c nodered.yml nodered

## nodered.yml
## node red + traefik v2 stack
## create traefik overlay network first
## forward port 80 to your swarm master node if using http challenge
## forward port 443 to your swarm for remote access
## no port forwards needed if using DNS challenge unless you need remote access
## node red will be available at https://nodered.yourdomain.tld
## deploy to a docker swarm, single node or more


services:

version: '3.7'

# TRAEFIK 2.2
# DO NOT USE LATEST

services:
  traefik:
    image: traefik:v2.2
    command:
      - "--log.level=DEBUG"  #CHANGE TO INFO AFTER SUCCESS
      - "--global.sendAnonymousUsage=true" 
      - "--accessLog=true"
      - "--accessLog.filePath=/var/log/docker/traefik.log"
      - "--accessLog.bufferingSize=100"
      - "--metrics.influxdb=false"
      - "--serversTransport.insecureSkipVerify=false"
      - "--api.dashboard=true"
      - "--providers.docker.endpoint=unix:///var/run/docker.sock"
      - "--providers.docker.swarmMode=true"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.docker.network=traefik-public"
      - "--providers.file.directory=/etc"
      - "--providers.file.watch=true"
      - "--entrypoints.web.address=:80"   #port 80 ingress optional or if you need http challenge
      - "--entrypoints.websecure.address=:443"    #port 443 ingress if you want remote access

# PICK ONE TYPE OF SSL CHALLENGE

# 1 LETSENCRYPT HTTP CHALLENGE
#      - "--certificatesresolvers.letsencryptresolver.acme.httpchallenge=true"
#      - "--certificatesresolvers.letsencryptresolver.acme.httpchallenge.entrypoint=web"

# LETSENCRYPT OPTIONS
#      - "--certificatesresolvers.letsencryptresolver.acme.email=your@email"
#      - "--certificatesresolvers.letsencryptresolver.acme.storage=/letsencrypt/acme.json"
#      - "--certificatesresolvers.letsencryptresolver.acme.caServer=https://acme-staging-v02.api.letsencrypt.org/directory"

# 2 CLOUDFLARE DNS CHALLENGE
      - "--certificatesresolvers.letsencryptresolver.acme.email=your@email"
      - "--certificatesresolvers.letsencryptresolver.acme.dnschallenge=true"
      - "--certificatesresolvers.letsencryptresolver.acme.dnschallenge.provider=cloudflare"
      - "--certificatesresolvers.letsencryptresolver.acme.dnschallenge.delaybeforecheck=90"
      - "--certificatesresolvers.letsencryptresolver.acme.dnsChallenge.resolvers=1.1.1.1:53,1.0.0.1:53"
      - "--certificatesresolvers.letsencryptresolver.acme.storage=/letsencrypt/acme.json"
	  # USE THE STAGING SERVER UNTIL YOU GET ISSUED A LETSENCRYPT STAGING CERT, THEN COMMENT BELOW LINE AND RUN AGAIN
      - "--certificatesresolvers.letsencryptresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"

# FOR CLOUDFLARE DNS CHALLENGE
    environment:
      - CF_API_EMAIL=your@email
      - CF_DNS_API_TOKEN=your_cloudflare_api_token
      - CF_ZONE_API_TOKEN=your_cloudflare_zone_token

    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host

    volumes:
      - traefik-certificates:/letsencrypt
      - /var/run/docker.sock:/var/run/docker.sock:ro

    networks:
      - traefik-public

    deploy:
      replicas: 1
      placement:
        constraints:
          - node.role == manager

#global redirect to https
      labels:
        - "traefik.http.routers.http-catchall.rule=hostregexp(`{host:.+}`)"
        - "traefik.http.routers.http-catchall.entrypoints=web"
        - "traefik.http.routers.http-catchall.middlewares=redirect-to-https"
        - "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
        

# NODE-RED
  nodered:
    image: nodered/node-red:latest-12
    volumes: 
      - nodered:/data
    environment:
      - TZ=America/Vancouver
    networks: 
      - traefik-public
      - bridge
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      labels:
      - "traefik.enable=true"
      - "traefik.http.routers.nodered.rule=Host(`nodered.yourdomain.tld`)" #insert your domain here
      - "traefik.http.routers.nodered.entrypoints=websecure"
      - "traefik.http.routers.nodered.tls=true"
      - "traefik.http.routers.nodered.tls.certresolver=letsencryptresolver"
      - "traefik.docker.network=traefik-public"
      - "traefik.http.services.nodered.loadbalancer.server.port=1880"
      - "traefik.http.middlewares.nodered.headers.SSLRedirect=true"
      - "traefik.http.middlewares.nodered.headers.STSSeconds=315360000"
      - "traefik.http.middlewares.nodered.headers.browserXSSFilter=true"
      - "traefik.http.middlewares.nodered.headers.contentTypeNosniff=true"
      - "traefik.http.middlewares.nodered.headers.forceSTSHeader=true"
      - "traefik.http.middlewares.nodered.headers.STSIncludeSubdomains=true"
      - "traefik.http.middlewares.nodered.headers.STSPreload=true"
      - "traefik.http.middlewares.nodered.headers.frameDeny=true"

volumes:
  nodered:
  traefik-certificates:
	  
networks:
  traefik-public:
    external: true
  bridge:
    external: true

your reward:

image
image866Ă—494 27.3 KB

2 Likes
2

Thanks @VinistoisR,
It is really impressive that this can all be done with a single Docker compose file!
Now I'm getting completely new thoughts about my Node-RED setup at home...
But some basic questions to get me started:

Just to understand the complete story: In my acme-client setup I had started listening to port 80 only for time required (to allow Letsencrypt to get the token). Once I had received the certificate, I stopped listening ... I had understood that it is not good practice to start Node-RED as admin user (to allow it to listen to port 80), to avoid that NodeJs has too much permissions (in case some maleficent people have control over your Node-RED). But some people claim it is insecure to have Node-RED listening to port 80 always (instead of port 1880), while others say it is no problem. Do you have any thoughts about that?

Is it required to have your own domain, or is this also possible via a (free) dynamic dns provider?

Do I understand this correctly:

  • the Node-RED instance which is public accessible, will be accessed from the browser via Traefik and then Docker forwards port 80 to that instance.
  • the other Node-RED instances which are not public accessible needs to run an another overlay network (or perhaps not in Docker?). But would it be worse if you add those also on the traefik-public network, but without a published port?

Lots of us have multiple Node-RED instances running (on Raspberries or whatever devices or cloud services). Is there an easy way to add such instances to your yml file also? I had always thought 'most' of us will need a Docker swarm to do load balancing of services, so I assume a simple overlay network will be the best to get started with? Do you have any thoughts about that?

Is it also possible to have multiple instances of Node-RED running, but with a different version? Suppose I need to test one of my nodes on NodeJs version 10 to solve an issue, while my production Node-RED system runs on NodeJs version 12. Do you have any idea whether something like that is possible?

Thanks for your time!!!!!!!!

3

I think the port numbers are kind of irrelevant, its more a question of allowing non encrypted traffic. These days you can do everything ssl over port 443. Personally I have my docker host listening on port 80, and immediately hits the traefik "global redirect to https" rule. Notice that "http-catchall" is the only router listening on the "web" entrypoint (port 80). I do this so that I can just type "nodered" into a browser, and it immediately resolves that to http://nodered.mydomain.tld, which hits my docker host and immediately upgrades to https://nodered.mydomain.tld. I do not, however, open port 80 on my firewall. So every external request coming in has to specify https://.

I don't really like http validation, there's a pretty vulnerable moment involved. I highly recommend users figure out DNS validation. Traefik supports tons of dns providers, even DuckDNS. But for a pain-free experience, point whatever you have to cloudflare. It's free.

super easy. Notice how I'm creating a new "router" in traefik, by declaring one that doesn't exist yet "traefik.http.routers.nodered2.rule=...". For all the traefik labels in the second service, change 'nodered' to 'nodered2'.


services:
  nodered1:
    labels:
     - "traefik.http.routers.nodered.rule=Host(`nodered.yourdomain.tld`)" #insert your domain here
     - etc etc

  nodered2:
    image: node-red:latest-10
    labels:
     - "traefik.http.routers.nodered2.rule=Host(`nodered2.yourdomain.tld`)" #insert your domain here
     - etc etc
  
  nodered3:
    image: node-red:dev
    labels:
     - "traefik.http.routers.nodered3.rule=Host(`nodered3.yourdomain.tld`)" #insert your domain here
     - etc etc

You never expose the nodered port out of its container. You don't expose any... Traefik has the back door key;)

What happens is the request comes into the docker host, traefik looks at the URL and figures out which rule matches. It figures out which service it is looking for, strips the SSL and sends along the un-encrypted packets to the container.

When you launch other containers to this stack, they can address node-red just by its service name. so instead of pointing something to http://172.16.1.1:1880/, you can just type nodered1:1880. and inside node red, you can connect to your other services in the same way:

image
image1171Ă—371 11.6 KB

notice I don't need to use https://, because I'm inside the docker stack. Docker sorts out the DNS. Doesn't matter where in my swarm my home-assistant container is running, and I certainly don't care what its ip address is.

So if you had multiple node-red containers defined, one could ping the other by typing "ping nodered2".

4

A couple of other tips:

  • Some routers will give you trouble trying to loop a request from the LAN back into the WAN. I use PFsense so their nat reflection guide was helpful. If you can access your services externally but not internally, NAT reflection is probably the culprit.

  • Traefik runs on your master node, but your containers can be on any node. Docker sorts out the details.

  • You don't necessarily need your services to share the same stack to address each other so easily. All they need is to share a common overlay network. It's no issue at all to create a network specifically for this. For example, I have a network that spans all my services that need a time-series database, and one influxdB container on that network. So across a bunch of different stacks, they can all reach the db at influxdb:8086.

  • You can mix architectures. I have an arm64 node in my swarm (I'm going to add more!). You can launch services with arm-specific images, and restrict them to arm nodes, and route traffic to them the same way. In fact I have some images that will happily run on either arm or x86 nodes. That, to me, is amazing.

  • Traefik can add its own basic auth to any service. But I much prefer oauth integration, which I'll share next :wink: Then you can just log in with your google creds.

  • Once your setup with cloudflare is working, you can enable their "Proxy Mode" and you get a whole other layer of security, including a firewall with geo-ip filtering, rate limiting, ddos protection, etc etc. However you need to be real with yourself that your public ip is still open.

  • The way my example sets up traefik in a single compose file is my preferred method. TOML files are another method. The way it is shown here, in the traefik docs click on 'CLI' in the examples to see the applicable commands.

5

Evening Vinistois,
This is way out of my comfort zone, but you are a good teacher :+1:
But the more I read, the more questions keep popping up ...

DNS01 challenge

If I try to map your explanation to some drawing I found here:

image
image966Ă—655 133 KB

  • So in this case Traefik (running on the Node-RED server) will create the CSR, or not? Because when I have multiple nodes, only the master node will have Traefik container. How do the other nodes trigger a CSR then?
  • Will Traefik also send an updated WAN address to the DNS provider perhaps? I do the latter one at the moment within my Node-RED flow.
  • What do they mean by control server? Is that also part of Traefik?
  • Do I understand correctly that you would use DNS-01 challenges both for public and private Node-RED servers (instead of http-01 challenge for public servers)?

Technical overview

I 'tried' to visualize your explanation:

image
image953Ă—416 88.9 KB

For me personally the Cloudflare network seems a bit overhead, since I don't think I need the worldwide caching / compression / ... So I'm going to start by keeping my Duckdns sub-domain forwarding my requests directly to my home modem(router).

  • You say that Traefik strips the SSL. Do you mean it does SSL termination, and that from here on all data through the overlay network(s) is unencrypted? So no SSL setup in Node-RED required anymore?
  • When I have multiple Docker host machines (e.g. raspberries ...), do I need to add such a Docker compose yml file on every machine? And then join those extra nodes manually to the swarm? Or do I have a single yml file that contains the information (e.g. IP addresses) of all machines?
  • Your yml file contains Traefik-related labels under the Node-RED service. Is that used for linking Traefik to Node-RED I assume?
  • When you have multiple Node-RED containers running on the same machine, do we need to use separate volumes? And where are those volumes located on the host system? Just wondering where I should copy/backup e.g. my Node-RED flow files.
  • You say that Traefik is only running on the master node, is that due to the "global" mode (instead of "replicated")?
  • I thought that a Docker swarm (with nodes) was only required if you wanted to have a load-balanced replication of the services? And that otherwise you could also create an overlay network across multiple machines without a swarm. Is that correct, or am I now getting nuts?
  • Is it better to have a second separate overlay network for all backend stuff (like your timeseries db)? Not sure whether that is required for security, or other reasons...

P.S. Hopefully you are not near a burnout now, because we are getting closer :rofl:

6

I'm sorry, those parts I have not looked into. I use Traefik as directed because all that stuff is in the weeds for me and not aligned with my core interest, which is just to make it work reliably with minimal maintenance. Set and forget.

For DDNS, that's more of a router job, imho. not something I'd be doing inside node-red.

DNS challenge requires nothing of your host. Traefik just plants the txt records at your domain and LE looks for them there. You can totally use DNS validation to provide an SSL cert for a container that is not exposed at all to the outside world. All you need is to park your domain at a DNS provider that's on Traefik's list of dns provider integrations.

Yes it's definitely extra. What I like is they add geo-ip headers to the incoming requests. I use those for a custom thing. I also use the firewall functions and the logging / tracing functions. When I hit my sites externally I'm served a cloudflare cert. When I hit internally (or over one of my vpns) I get the lets encrypt one.

Thats right, it terminates the ssl and forwards un-encrypted http to the service. My compose file above works with a default install of node-red. I usually have to restart the stack at least once, be patient, and hard refresh the browser to get the valid SSL cert. In the past, when I've had trouble, it was due to rushing the process.

First, switch to swarm mode whether you have one host or more. Portainer has a limitation with compose file versions and features when it's not in swarm mode. There's no reason not to.

Once you are in swarm mode you can use docker stack deploy on any master node. So your actual compose file can live wherever you like, you just have to specify its path with the -c flag:

docker stack deploy -c /path/to/mystack.yml mystack

or don't keep the file locally:

docker login -u username -p password registry.hub.Docker.com/myproject && Docker stack deploy -c Docker-swarm.yml test --with-registry-auth

personally the only stack I deploy on the command line is portainer. Once portainer is up and running I deploy new stacks from the GUI. That way you end up with a stack that portainer can manage. Deploying stacks via portainer and command line is the same, you just paste the compose yml right into the 'deploy stack' box.

Yes, that's how you tell Traefik what to do with this service. Traefik picks up those labels on the fly (you don't have to restart traefik) for new services that launch. It gets this info from its hook into the docker daemon.

yes each will need its own volume.

If you create a named volume at the service level, you have to define it at the stack level (I put my stack volume declarations at the bottom, with the network ones)

Here you have choices:

volumes:
  node-red:

This will create a "docker named volume". When you start this stack, whatever node this service is running on will have a dir created wherever docker puts its default volumes... /var/lib/docker/volumes/node-red/_data/ on my hosts. But there is no mechanism to keep those folders in sync. When you move this service to another node (or it restarts on a new node), the data is not transferred from one to the other, so it will just start fresh. If you have one node, this doesn't matter. If you have multiple, you need to figure out shared storage.

So if you supply:

  nodered:
    driver: local
    name: nodered
    driver_opts:
      type: none
      o: bind
      device: /mnt/swarm/nodered

now you are telling docker to bind mount a local directory to that named volume, instead. The contents of /mnt/swarm/nodered had better be the same on every host you launch this on!! I use NFS for that. You can use whatever kind of distributed storage you want, and mount it into /mnt/swarm/ (for example). That way no matter which host the service is started on, when it bind-mounts /mnt/swarm/node-red/, the contents are the same.

If the dir you specify is empty when the service first starts, it will copy files into it according to the dockerfile. If the dir already contains files, then it will be bind-mounted as is. (Careful with this...). That's useful for recoveringing or transferring installs from one setup to another.

I'm sorry, the "global" mode is not what you should be using. I am using it for a custom deployment which is beyond the scope of this conversation. You should restrict Traefik to one container on one master node, using replicas: 1

IIRC overlay networks are a swarm thing.

I do it for the same reason I do vlans... segregates traffic, removes barriers between services that need to talk, and implies barriers for ones that don't.

1 Like
7

this was posted today, it's a great resource, very similar to my deployments. Between my example and the ones here, you should have this sorted out.

2 Likes
8

When I read you,
I don't understand much because I don't have Docker's experience.
Would it be possible to have a schematic of all the components, a view of the infrastructure needed to run this automatic certificate management via Traefik?
How many VMs or physical machines, where are the Docker, NodeRed, Mosquitto?

I have the feeling that with this solution, we can make H.A with our home automation and it's very interesting!

Bart's original idea of a Nodered node that manages this function automatically suited me well.

9

I certainly agree here. I use cloudflare for all of my domains now. It provides visibility of who is accessing things, an extra layer of security (potentially several depending on configuration) and considerable protection from DDOS attacks (which might be random or might be hiding other attacks). I only use their free services though I do now have some domains actually registered via them (they don't support all UK-specific domains as yet).

10

Jean-Luc,
you can find above my draft schematic of yesterday. Will try to update it tonight with all extra comments that Vinistois has added today.

I'm not sure at the moment if I need to send my acme-client node straight to the garbage bin, or that it can perhaps be of any use after all. Because like you say, I don't think all our users are ready to switch to Docker. Would be nice if we could support both user groups... Of course I don't want to create a highly insecure node.

1 Like
11

I would just add here that there is a lot of push back in Linux land at the moment against docker and running as an elevated daemon. Various distros are coming out with alternative approaches for container lifecycle management and running that get away from the need for an elevated daemon to run them - for example look at Podman under RedHat and Centos for the way forward on that side of the fence.

Craig

1 Like
12

That is one of the things that is concerning me, I am about to build a new server and was considering this approach, but the last thing I want is to learn about it, implement it, and then in a couple of years have to re-do it all as the technique has gone out of favour.

13

The technique is here to stay and learning and implementing containers will hold you in good stead for the next 5 years or so - it will just be a maturing landscape.

All of the proposed techniques etc translate pretty well across most of the container implementations and once you understand the basic technqiues you will be able to reapply them to any of the management systems.

The new Podman system in Centos is almost a direct drop in for Docker and will use 99% of the yaml files to create and manage the same containers etc

Craig

14

OK, thanks Craig, I will have to have a look. Being retired one has so little time though.
You may think that doesn't make sense but if you are retired you will know exactly what I mean, and if you are not retired then just wait, you will see.

2 Likes
15

Would like to thank @VinistoisR for all the time he spend on this discussion!!

I think that Craig is preparing us for his upcoming tutorial "Running Node-RED in non-Docker containers" :wink:

1 Like
16

Ha Ha - not retired yet - but i know what you mean.

My parents went through the same thing a little while ago and then decided that they had to treat retirement like a job (that they had for the rest of thier lives) - so they now right a daily todo/tasklist and map things out on an hourly basis that they want to get done each day - its pretty amazing the difference it has made in terms of their outlook on life

Craig

17

I was thinking of redoing my setup into Centos 8 and Podman so i might just do that over the next month or so Bart !! Thanks for prompting me !

Craig

1 Like
18

Compose has now become an open standard specification and is platform-agnostic:

People have been saying docker is going away since it gained popularity. Go on Google trends, put in docker, Kubernetes, and podman. Look at the 5 year trend. Docker isn't going anywhere.

19

Yes i am aware of the Compose-Spec - my point was more that due to the (by Design) insecure nature of the way that Docker runs - there are many alternatives coming out - hopefully Kubernetes and Compose will gain enough traction to act as the unifying foce regardless of what container runtime is used.

Craig

20

Yep! We're in agreement. Although there's essentially no security risk in an ssl-fronted local service that requires no firewall ports open. That's what I'm suggesting here. Node-red in docker behind an ssl-terminating web proxy is much, much more secure than a bare install with a port forward.