Dashboard failing on kubernetes cluster

Hi All, we are trying to figure out how to solve a problem... Stack is Google Kubernetes Engine + nginx reverse proxy + node red in docker pods deployed on the cluster consisting of GCP high mem servers.

We are seeing the dashboard 404ing after flows are redeployed. Sometimes without any actual changes being made to the flows beyond creating a new blank flow, for instance. The dashboard comes back to life if we restart Node Red.

The admin/flow interface is fine, any endpoints (for instance url.com/worldmap) are fine - all keep running. It's just the dashboard.

We are seeing nothing in the logs - even on trace debug, which are giving us any clues as to what is going on and where it is failing either on the Node Red logs or any of the container or Kubernetes logs.

Memory usage is tiny (less than 5%, CPU usage likewise).

Struggling to figure out where to look next!

I know there's a bunch of info missing here, for reasons, but I just wanted to ask out if anyone else has had any experience of this kind of thing happening on a similar stack? It's driving us potty!

Best wishes

Dan

Looks like this is not something anyone else has encountered? @knolleary (sorry to tag you in directly) don't suppose you have seen any clues as to any directions to take here to solve these issues? I am wondering if there is something fundamental that we have misunderstood about the architecture of NR and/or the Dashboard which means we are going to struggle with this?

Any help gratefully received!

Ah! We appear to have found the issue and a potential solution, though it does not resolve what we wanted to do originally...

We had the dashboard root set to:
settings.ui.path = ''

as we wanted to have the ui at the root of each subdomain. We have put the settings back to

settings.ui.path = '/ui'

and that seems to have helped stop the 404s...

Only trouble is, we are now getting a redirect error, possibly because we have also enabled HTTP static serving and put an index.html at the root with a redirect to /ui... But we shall attempt to fix this as well. Just wanted to update anyone reading this in the meantime.

It is obvious now that this ought to be in the Dashboard category... not sure how to move the thread there... but hopefully there are some helpful mods out there - and not the type that ride scooters and listen to The Who! :wink:

Further updates when we get them. If anyone knows why this is happening, I'd still love to hear from them before we start digging into Github and trying to work it out!

We are using this exact architecture, but our problem is with securing the editor and ui. In a cluster, the session cookie issued by one node is not understood by the others as the LB round-robins requests to them. We need to explore some kind of shared session store...

I think we resolved that issue...

I will have a chap with my Über Dev and see what he says.

d.

1 Like

Have you tried setting settings.ui.path to / rather than a blank string?

I had thought of that. Will try in the morning. See if it helps. Thanks for responding Nick. This has been a really troublesome issue!

Ah. Yes. We did try this '/' option... and it failed still. I will raise an issue on github and we can see what we can do to work out what the problem is.