Node-RED SSL reverse proxy w/ Google Oauth

VinistoisR · 2 May 2020 18:12

Ok, try now. I forgot the config declaration in the stack. I don't think you did anything wrong. Thanks for spotting the typo! Btw I load in all my configs through the portainer UI, which only has one field for name, there isn't a separate filename. If you are still having troubles, try that way. I'll work on converting that toml so we can just have it in the static labels, instead of a separate file. I just found it easier at the time to copy the toml into a config.

You've probably deduced by now that .toml files are kind of the same as attaching labels to traefik in the compose file, it's two ways to do the same thing, just with different syntax.

Notice this in the traefik config:

      - "--providers.file.directory=/etc"
      - "--providers.file.watch=true"

That means you can drop any valid toml file into /etc/ in the container and traefik will pick it up on the fly. I use config files to get them there, you can just as easily bind files into /etc/ if you prefer. This is useful for adding services behind traefik that aren't even on your docker host. So if you already have some kind of service with no ssl on your network, a quick toml definition put into /etc/ will proxy it as if it were a docker service, without even restarting anything.

Colin · 2 May 2020 21:24

Great, thanks. Much better now, I have got as far as getting the staging certificate (and probably the real one but it hasn't worked its way to the browser yet. However, I am seeing, in the log

time="2020-05-02T21:19:34Z" level=error msg="Router defined multiple times with different configurations in [oatraefik-nodered-oox0sjblssej3pcvdh9kq71zc oatraefik-traefik-49q9e9v3d1bo3bge2lohvt75r]" providerName=docker routerName=traefik

It's getting late here now, so have to leave it for the moment.

VinistoisR · 2 May 2020 23:56

silly mistake on my end. My error was at the bottom of the node-red service declaration, I referenced the traefik router instead of the node-red router.

Colin · 3 May 2020 10:05

Excellent, it is all working now.
One minor points, the oauth service has restart: always, it should be using restart_policy. Should it be set to on-failure as for node-red or left at default (which is any I think) as for traefik?

Also one issue, which may be normal behaviour, with the logs on debug I see the log below, repeated about every 15 seconds, as if something is repeatedly restarting. Though as I said, this may be normal.

time="2020-05-03T09:40:42Z" level=debug msg="Network not found, id: s79msp9gdl2xbppxdk0bfhudx" providerName=docker


time="2020-05-03T09:40:42Z" level=debug msg="Filtering disabled container" providerName=docker container=portainer-portainer-qr5sa6fkhw5j45c6tu7vxc8ls


time="2020-05-03T09:40:42Z" level=debug msg="Configuration received from provider docker: {\"http\":{\"routers\":{\"http-catchall\":{\"entryPoints\":[\"web\"],\"middlewares\":[\"redirect-to-https\"],\"service\":\"dummy-svc\",\"rule\":\"hostregexp(`{host:.+}`)\"},\"nodered\":{\"entryPoints\":[\"websecure\"],\"middlewares\":[\"admin\"],\"service\":\"nodered\",\"rule\":\"Host(`nodered.mydomain.org.uk`)\",\"tls\":{\"certResolver\":\"letsencryptresolver\"}},\"oauth-rtr\":{\"entryPoints\":[\"websecure\"],\"middlewares\":[\"oauth@file\"],\"service\":\"oauth-svc\",\"rule\":\"Host(`auth.mydomain.org.uk`)\",\"tls\":{\"certResolver\":\"letsencryptresolver\"}},\"traefik\":{\"entryPoints\":[\"websecure\"],\"middlewares\":[\"admin\"],\"service\":\"api@internal\",\"rule\":\"Host(`traefik.mydomain.org.uk`)\",\"tls\":{\"certResolver\":\"letsencryptresolver\"}}},\"services\":{\"dummy-svc\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.2.35:9999\"}],\"passHostHeader\":true}},\"nodered\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.2.40:1880\"}],\"passHostHeader\":true}},\"oauth-svc\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.0.2.38:4181\"}],\"passHostHeader\":true}}},\"middlewares\":{\"admin\":{\"chain\":{\"middlewares\":[\"me-only\",\"oauth@file\"]}},\"me-only\":{\"ipWhiteList\":{\"sourceRange\":[\"192.168.0.0/16\",\"210.45.23.241/16\"]}},\"nodered\":{\"headers\":{\"sslRedirect\":true,\"stsSeconds\":315360000,\"stsIncludeSubdomains\":true,\"stsPreload\":true,\"forceSTSHeader\":true,\"frameDeny\":true,\"contentTypeNosniff\":true,\"browserXssFilter\":true}},\"redirect-to-https\":{\"redirectScheme\":{\"scheme\":\"https\"}}}},\"tcp\":{},\"udp\":{}}" providerName=docker


time="2020-05-03T09:40:42Z" level=debug msg="Adding certificate for domain(s) vps02.mydomain.org.uk"


time="2020-05-03T09:40:42Z" level=debug msg="Adding certificate for domain(s) auth.mydomain.org.uk"


time="2020-05-03T09:40:42Z" level=debug msg="Adding certificate for domain(s) traefik.mydomain.org.uk"


time="2020-05-03T09:40:42Z" level=debug msg="Adding certificate for domain(s) nodered.mydomain.org.uk"


time="2020-05-03T09:40:42Z" level=debug msg="No default certificate, generating one"


time="2020-05-03T09:40:42Z" level=debug msg="Creating middleware" entryPointName=web routerName=http-catchall@docker serviceName=dummy-svc middlewareName=pipelining middlewareType=Pipelining


time="2020-05-03T09:40:42Z" level=debug msg="Creating load-balancer" entryPointName=web routerName=http-catchall@docker serviceName=dummy-svc


time="2020-05-03T09:40:42Z" level=debug msg="Creating server 0 http://10.0.2.35:9999" routerName=http-catchall@docker serviceName=dummy-svc serverName=0 entryPointName=web


time="2020-05-03T09:40:42Z" level=debug msg="Added outgoing tracing middleware dummy-svc" middlewareName=tracing entryPointName=web routerName=http-catchall@docker middlewareType=TracingForwarder


time="2020-05-03T09:40:42Z" level=debug msg="Creating middleware" middlewareType=RedirectScheme entryPointName=web routerName=http-catchall@docker middlewareName=redirect-to-https@docker


time="2020-05-03T09:40:42Z" level=debug msg="Setting up redirection to https " entryPointName=web routerName=http-catchall@docker middlewareName=redirect-to-https@docker middlewareType=RedirectScheme


time="2020-05-03T09:40:42Z" level=debug msg="Adding tracing to middleware" entryPointName=web routerName=http-catchall@docker middlewareName=redirect-to-https@docker


time="2020-05-03T09:40:42Z" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery


time="2020-05-03T09:40:42Z" level=debug msg="Creating Middleware (ResponseModifier)" routerName=nodered@docker middlewareName=admin@docker middlewareType=Chain entryPointName=websecure


time="2020-05-03T09:40:42Z" level=debug msg="Creating middleware" entryPointName=websecure routerName=nodered@docker middlewareName=pipelining middlewareType=Pipelining serviceName=nodered


time="2020-05-03T09:40:42Z" level=debug msg="Creating load-balancer" entryPointName=websecure routerName=nodered@docker serviceName=nodered


time="2020-05-03T09:40:42Z" level=debug msg="Creating server 0 http://10.0.2.40:1880" serviceName=nodered serverName=0 entryPointName=websecure routerName=nodered@docker


time="2020-05-03T09:40:42Z" level=debug msg="Added outgoing tracing middleware nodered" middlewareName=tracing middlewareType=TracingForwarder entryPointName=websecure routerName=nodered@docker


time="2020-05-03T09:40:42Z" level=debug msg="Creating middleware" middlewareName=admin@docker middlewareType=Chain routerName=nodered@docker entryPointName=websecure


time="2020-05-03T09:40:42Z" level=debug msg="Creating middleware" middlewareName=oauth@file middlewareType=ForwardedAuthType entryPointName=websecure routerName=nodered@docker


time="2020-05-03T09:40:42Z" level=debug msg="Adding tracing to middleware" routerName=nodered@docker middlewareName=oauth@file entryPointName=websecure


time="2020-05-03T09:40:42Z" level=debug msg="Creating middleware" routerName=nodered@docker middlewareName=me-only@docker middlewareType=IPWhiteLister entryPointName=websecure


time="2020-05-03T09:40:42Z" level=debug msg="Setting up IPWhiteLister with sourceRange: [192.168.0.0/16 210.45.23.241/16]" entryPointName=websecure routerName=nodered@docker middlewareName=me-only@docker middlewareType=IPWhiteLister


time="2020-05-03T09:40:42Z" level=debug msg="Adding tracing to middleware" routerName=nodered@docker entryPointName=websecure middlewareName=me-only@docker


time="2020-05-03T09:40:42Z" level=debug msg="Creating middleware" routerName=oauth-rtr@docker serviceName=oauth-svc entryPointName=websecure middlewareName=pipelining middlewareType=Pipelining


time="2020-05-03T09:40:42Z" level=debug msg="Creating load-balancer" entryPointName=websecure routerName=oauth-rtr@docker serviceName=oauth-svc


time="2020-05-03T09:40:42Z" level=debug msg="Creating server 0 http://10.0.2.38:4181" serverName=0 entryPointName=websecure routerName=oauth-rtr@docker serviceName=oauth-svc


time="2020-05-03T09:40:42Z" level=debug msg="Added outgoing tracing middleware oauth-svc" middlewareName=tracing middlewareType=TracingForwarder entryPointName=websecure routerName=oauth-rtr@docker


time="2020-05-03T09:40:42Z" level=debug msg="Creating middleware" entryPointName=websecure routerName=oauth-rtr@docker middlewareName=oauth@file middlewareType=ForwardedAuthType


time="2020-05-03T09:40:42Z" level=debug msg="Adding tracing to middleware" entryPointName=websecure routerName=oauth-rtr@docker middlewareName=oauth@file


time="2020-05-03T09:40:42Z" level=debug msg="Creating Middleware (ResponseModifier)" middlewareName=admin@docker middlewareType=Chain entryPointName=websecure routerName=traefik@docker


time="2020-05-03T09:40:42Z" level=debug msg="Added outgoing tracing middleware api@internal" entryPointName=websecure routerName=traefik@docker middlewareName=tracing middlewareType=TracingForwarder


time="2020-05-03T09:40:42Z" level=debug msg="Creating middleware" routerName=traefik@docker middlewareName=admin@docker middlewareType=Chain entryPointName=websecure


time="2020-05-03T09:40:42Z" level=debug msg="Creating middleware" entryPointName=websecure routerName=traefik@docker middlewareName=oauth@file middlewareType=ForwardedAuthType


time="2020-05-03T09:40:42Z" level=debug msg="Adding tracing to middleware" routerName=traefik@docker entryPointName=websecure middlewareName=oauth@file


time="2020-05-03T09:40:42Z" level=debug msg="Creating middleware" middlewareName=me-only@docker middlewareType=IPWhiteLister entryPointName=websecure routerName=traefik@docker


time="2020-05-03T09:40:42Z" level=debug msg="Setting up IPWhiteLister with sourceRange: [192.168.0.0/16 210.45.23.241/16]" entryPointName=websecure routerName=traefik@docker middlewareName=me-only@docker middlewareType=IPWhiteLister


time="2020-05-03T09:40:42Z" level=debug msg="Adding tracing to middleware" routerName=traefik@docker middlewareName=me-only@docker entryPointName=websecure


time="2020-05-03T09:40:42Z" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery


time="2020-05-03T09:40:42Z" level=debug msg="Try to challenge certificate for domain [auth.mydomain.org.uk] found in HostSNI rule" providerName=letsencryptresolver.acme rule="Host(`auth.mydomain.org.uk`)" routerName=oauth-rtr@docker


time="2020-05-03T09:40:42Z" level=debug msg="Try to challenge certificate for domain [traefik.mydomain.org.uk] found in HostSNI rule" providerName=letsencryptresolver.acme routerName=traefik@docker rule="Host(`traefik.mydomain.org.uk`)"


time="2020-05-03T09:40:42Z" level=debug msg="Try to challenge certificate for domain [nodered.mydomain.org.uk] found in HostSNI rule" providerName=letsencryptresolver.acme rule="Host(`nodered.mydomain.org.uk`)" routerName=nodered@docker


time="2020-05-03T09:40:42Z" level=debug msg="Looking for provided certificate(s) to validate [\"nodered.mydomain.org.uk\"]..." routerName=nodered@docker providerName=letsencryptresolver.acme rule="Host(`nodered.mydomain.org.uk`)"


time="2020-05-03T09:40:42Z" level=debug msg="No ACME certificate generation required for domains [\"nodered.mydomain.org.uk\"]." rule="Host(`nodered.mydomain.org.uk`)" routerName=nodered@docker providerName=letsencryptresolver.acme


time="2020-05-03T09:40:42Z" level=debug msg="Looking for provided certificate(s) to validate [\"auth.mydomain.org.uk\"]..." providerName=letsencryptresolver.acme rule="Host(`auth.mydomain.org.uk`)" routerName=oauth-rtr@docker


time="2020-05-03T09:40:42Z" level=debug msg="No ACME certificate generation required for domains [\"auth.mydomain.org.uk\"]." providerName=letsencryptresolver.acme rule="Host(`auth.mydomain.org.uk`)" routerName=oauth-rtr@docker


time="2020-05-03T09:40:42Z" level=debug msg="Looking for provided certificate(s) to validate [\"traefik.mydomain.org.uk\"]..." rule="Host(`traefik.mydomain.org.uk`)" providerName=letsencryptresolver.acme routerName=traefik@docker


time="2020-05-03T09:40:42Z" level=debug msg="No ACME certificate generation required for domains [\"traefik.mydomain.org.uk\"]." routerName=traefik@docker rule="Host(`traefik.mydomain.org.uk`)" providerName=letsencryptresolver.acme

VinistoisR · 3 May 2020 11:42

That's normal, just set it to INFO.

Colin · 3 May 2020 12:29

Great, thanks.
Mosquitto next, plus working out how to git clone into the node-red container as it is built.

Colin · 3 May 2020 15:36

I am struggling here, I can't work out how to get Traefik to route mqtt to mosquitto, and can't find anything in the docs to help. @VinistoisR, you mentioned mqtt earlier in the thread, have you used Traefik for this or is your mqtt not exposed to the internet?

VinistoisR · 3 May 2020 17:08

I use vernemq through traefik. Tcp routers are new in traefik, you'll need to get familiar with the docs.

Colin · 3 May 2020 19:18

OK, thanks. So much to learn, so little time for learning.
I will report back if I achieve success.

VinistoisR · 3 May 2020 21:06

The reason I use VerneMQ is it gives me zero trouble doing mqtt over wss (secure websocket) via traefik. It's the most basic config - just copy the labels from the nodered service and change all the "nodered" into "mqtt", and you'll be able to make websocket connection to:

wss://mqtt.yourdomain.co:443/mqtt

... Without messing with any cert files at all.

Put it in the same stack as nodered and then from nodered you can connect to mqtt:1883.

Colin · 3 May 2020 21:23

Do I put 1883 in
"traefik.http.services.mqtt.loadbalancer.server.port=1883"?

VinistoisR · 3 May 2020 21:52

No, you can't mqtt into an http port. Traefik http routers can only shuttle packets from http(s) entry points to http(s) backends. Mqtt is a TCP protocol, so you either need to use a TCP router, or publish the mqtt port (1883) and bypass traefik for mqtt connections.

If you use VerneMQ or another broker that supports mqtt over wss, then you can use traefik for that. Use your "websecure" entrypoint just like any other web service, and for loadbalancer.server.port, use whatever you have set as the wss port. IIRC VerneMQ listens on 8080 if you don't specify any different. A small gotcha with VerneMQ is you have to hit the /mqtt path, so your Traefik rule needs to catch that:

#VerenMQ Secure Websocket Proxy
       - "traefik.http.routers.vernemq_wss.rule=Host(`vernemq.mydomain.net`) && PathPrefix(`/mqtt`)"  # detect mqtt over websockets
       - "traefik.http.routers.vernemq_wss.entrypoints=websecure"
       - "traefik.http.routers.vernemq_wss.tls=true"
       - "traefik.http.routers.vernemq_wss.tls.certresolver=letsencryptresolver"
       - "traefik.http.routers.vernemq_wss.service=wss"
       - "traefik.http.services.wss.loadbalancer.server.port=8080"
       - "traefik.http.routers.vernemq_wss.middlewares=sslheader@docker"

Colin · 4 May 2020 08:41

I have got it working with vernemq and wss with some additional input from the second post on the link below, obviously posted by a very knowledgeable and helpful guy
There are a couple of small issues that I might need to come back on but it is basically working.
https://gist.github.com/Dacturne/80f09adc83080cd3b17eacf22baaf968

Colin · 5 May 2020 08:13

In case anyone is interested below is what is working for me with Vernemq over secure web sockets. To access it from a remote mqtt client such as node red, reference the server as wss://vernemq.mydomain.net:443/mqtt. To reference it locally from node red running in the same Traefik stack use Server vernemq Port 1883.

Many thanks again to @VinistoisR for his help getting this working.

 # VERNEMQ 
  vernemq:
    image: vernemq/vernemq:latest
    environment:
     - DOCKER_VERNEMQ_SWARM=1
     - DOCKER_VERNEMQ_ACCEPT_EULA=yes
     - DOCKER_VERNEMQ_ALLOW_ANONYMOUS=on		# set to off when working and users configured
     - DOCKER_VERNEMQ_PLUGINS.vmq_passwd=on
    volumes:
     - verne-config:/vernemq/etc
     - verne-data:/vernemq/data
    networks:
     - traefik-public
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.role == manager
      restart_policy:
        condition: on-failure
      labels:
#VerneMQ Status Dashboard at vernemq.mydomain.net/status
       - "traefik.enable=true"
       - "traefik.http.routers.vernemq.rule=Host(`vernemq.mydomain.net`) && PathPrefix(`/status`)"  
       - "traefik.http.routers.vernemq.entrypoints=websecure"
       - "traefik.http.routers.vernemq.tls=true"
       - "traefik.http.routers.vernemq.tls.certresolver=letsencryptresolver"  #your cert resolver must be setup in traefik already
       - "traefik.docker.network=traefik-public"
       - "traefik.http.routers.vernemq.service=vernemq"
       - "traefik.http.services.vernemq.loadbalancer.server.port=8888" #default http listener port
       - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https"   #needed to get wss working
       - "traefik.http.routers.vernemq.middlewares=sslheader@docker"
       
#VerenMQ Secure Websocket Proxy - wss://vernemq.mydomain.net/mqtt on port 443
#The /mqtt prefix is used for traefik and is also required for vernemq
       - "traefik.http.routers.vernemq_wss.rule=Host(`vernemq.mydomain.net`) && PathPrefix(`/mqtt`)"  # detect mqtt over websockets
       - "traefik.http.routers.vernemq_wss.entrypoints=websecure"
       - "traefik.http.routers.vernemq_wss.tls=true"
       - "traefik.http.routers.vernemq_wss.tls.certresolver=letsencryptresolver"
       - "traefik.http.routers.vernemq_wss.service=wss"
       - "traefik.http.services.wss.loadbalancer.server.port=8080"  #default vernemq ws port
       - "traefik.http.routers.vernemq_wss.middlewares=sslheader@docker"

VinistoisR · 5 May 2020 08:43

Glad you got it working

I'll template it out and add it to my repo.

Colin · 5 May 2020 10:44

It is virtually the same as you wrote in the link I posted, just with some unneeded bits trimmed off.

Colin · 5 May 2020 20:09

@VinistoisR, with the oauth google authorisation, how does one logout? I can't see any reference to that in the docs.

VinistoisR · 5 May 2020 20:21

Just log out of your Google account. Or, you can click the padlock in your browser, usually brings up the set cookies with a button "remove". Once the cookie is removed you can refresh the page and it will ask you to log back in.

Colin · 6 May 2020 07:54

That works, thanks.

Colin · 10 May 2020 13:25

@VinistoisR how should I restart node-red when the need arises? I tried restarting the container in Portainer and everything looked ok, including the node red startup log, but I can't access it, I get 404 not found. The only way I have found so far is by doing a full stack deploy.

Topic		Replies	Views
Behind Traefik proxy, cert shows valid but still insecure? General	5	1575	10 July 2019
Anyone tried traefik & docker node-red instances? General	3	2622	4 September 2019
Reverse proxy for node red with nginx and secure link setup? General nginx	11	726	15 December 2022
[ANNOUNCE] node-red-contrib-letsencrypt : beta version Share Your Nodes	65	4010	22 June 2020
Please Help! SSL , Let's Encrypt , digitalOcean , Https , 443 General	4	790	21 April 2021

Node-RED SSL reverse proxy w/ Google Oauth

Related Topics