Hi, I'm having a problem that I've been trying to solve for a few days now....but cannot seem to get anywhere with it...I have been using Nodered now for close to a year, and never had problems like i am having now 
TL;DR
- Trouble starting a Nodered Docker Container via docker stack deploy
- Started I Think since updating node-red-dashboard to version 2.19.4
- Now unable to start the nodered container at all via docker deploy commands
- Have tried numerous ways to tackle the problem but it still persists
Setup
(8x) Cluster of Raspberry Pi 4 -(Hostnames: p1, p2, p3...etc...p7, p8) all running Raspbian Buster Lite
[ Since problems have started I am only deploying on a single node - p1 (docker manager) ]
Running Docker v.19.03.8
Node-red image: nodered/node-red:1.0.4-12-arm32v7
Node-red-dashboard@2.19.4
Basically, this setup has been running fine for months (actually using dashboard v2.19.3) until I recently tried to update the dashboard node to v.2.19.4. Now my Nodered container will not start at all...
I am bind mounting the host directory /data/nodered/data to the container volume
The directory </data/nodered/data> contains...
pi@p1:/data/nodered/data $ ls -aln
total 90
drwxr-xr-x 5 1000 1000 4096 Mar 30 23:56 .
drwxr-xr-x 9 1000 1000 4096 Mar 30 03:15 ..
-rw-r--r-- 1 0 0 15284 Mar 30 23:56 .config.json
-rw-r--r-- 1 0 0 15285 Mar 30 22:46 .config.json.backup
-rw-r--r-- 1 0 0 483 Mar 30 22:19 flows.json
drwxr-xr-x 3 0 0 4096 Mar 30 21:26 lib
drwxr-xr-x 70 1000 1000 4096 Mar 30 22:44 node_modules
-rw-r--r-- 1 0 0 232 Mar 30 22:44 package.json
-rw-r--r-- 1 1000 1000 26776 Mar 30 22:45 package-lock.json
-rw-r--r-- 1 0 0 12478 Mar 30 21:26 settings.js
drwxr-xr-x 2 1000 1000 4096 Mar 30 22:07 tmp
^ Sorry if the formatting messes up here, i will try and fix it if it does..
My compose.yml file for docker stack deploy.....contains;
services:
nodered:
image: nodered/node-red:1.0.3-4-12-arm32v7
# image: nodered/node-red:1.0.3-4-12-minimal-arm32v7
# image: nodered/node-red:1.0.4-12-arm32v7
# image: nodered/node-red:1.0.4-12-minimal-arm32v7
# entrypoint: "npm start -- --userDir /data --safe"
ports:
- "1880:1880"
user: "0"
# user: "1000:1000"
environment:
TZ: "Europe/London"
deploy:
mode: replicated
replicas: 1
restart_policy:
condition: any
delay: 25s
max_attempts: 5
window: 90s
placement:
constraints:
- node.hostname==p1
volumes:
- "/data/nodered/data/:/data/"
- "/etc/timezone:/etc/timezone:ro"
- "/etc/localtime:/etc/localtime:ro"
The logs i am getting via docker logs -f stack_nodered are not very helpful.
I ahve also set the log level to 'trace' but there's still not much to deliver there
pi@p1:~ $ docker service logs -f stack_nodered
stack_nodered.1.hrljun5ah6oh@p1 |
stack_nodered.1.hrljun5ah6oh@p1 | > node-red-docker@1.0.3 start /usr/src/node-red
stack_nodered.1.hrljun5ah6oh@p1 | > node $NODE_OPTIONS node_modules/node-red/red.js $FLOWS "--userDir" "/data"
stack_nodered.1.hrljun5ah6oh@p1 |
stack_nodered.1.hrljun5ah6oh@p1 | 30 Mar 23:25:06 - [info]
stack_nodered.1.hrljun5ah6oh@p1 |
stack_nodered.1.hrljun5ah6oh@p1 | Welcome to Node-RED
stack_nodered.1.hrljun5ah6oh@p1 | ===================
stack_nodered.1.hrljun5ah6oh@p1 |
stack_nodered.1.hrljun5ah6oh@p1 | 30 Mar 23:25:06 - [info] Node-RED version: v1.0.3
stack_nodered.1.hrljun5ah6oh@p1 | 30 Mar 23:25:06 - [info] Node.js version: v12.15.0
stack_nodered.1.hrljun5ah6oh@p1 | 30 Mar 23:25:06 - [info] Linux 4.19.97-v7l+ arm LE
stack_nodered.1.hrljun5ah6oh@p1 | 30 Mar 23:25:06 - [info] Loading palette nodes
stack_nodered.1.xcygutoh0ga2@p1 |
stack_nodered.1.xcygutoh0ga2@p1 | > node-red-docker@1.0.3 start /usr/src/node-red
stack_nodered.1.xcygutoh0ga2@p1 | > node $NODE_OPTIONS node_modules/node-red/red.js $FLOWS "--userDir" "/data"
stack_nodered.1.xcygutoh0ga2@p1 |
stack_nodered.1.xcygutoh0ga2@p1 | 30 Mar 23:27:09 - [info]
stack_nodered.1.xcygutoh0ga2@p1 |
stack_nodered.1.xcygutoh0ga2@p1 | Welcome to Node-RED
stack_nodered.1.xcygutoh0ga2@p1 | ===================
stack_nodered.1.xcygutoh0ga2@p1 |
stack_nodered.1.xcygutoh0ga2@p1 | 30 Mar 23:27:09 - [info] Node-RED version: v1.0.3
stack_nodered.1.xcygutoh0ga2@p1 | 30 Mar 23:27:09 - [info] Node.js version: v12.15.0
stack_nodered.1.xcygutoh0ga2@p1 | 30 Mar 23:27:09 - [info] Linux 4.19.97-v7l+ arm LE
stack_nodered.1.xcygutoh0ga2@p1 | 30 Mar 23:27:10 - [info] Loading palette nodes
I know the logs show the image for version 1.0.3 - but it's exactly the same for v.1.0.4
The log just repeats and repeats without much useful information.
If i change the settings.js so include trace debugging, then the log shows the palette nodes being loaded - it reaches up to 'node-red-tail' before it bums out and restarts. But i know that node package comes pre-installed with node-red, so that cant be the problem.
Things I have Tried
- I have tried to use the --safe flag to start without the flows running
In docker compose...
entrypoint: "npm start -- --userDir /data --safe"
-- This also doesnt start - telling me my flow.json file is probably OK
-
I have tried using several images - 1.0.4....1.0.3....1.0.4-minimal......1.0.3-minimal
-- Same thing happens - so it cant be the image version -
I have tried $ 'docker exec -it <> /bin/bash' into the container so I can npm install node-red-dashboard@2.19.3 manually,
-- When the container is restarted it crashes and never starts. -
I removed node-red-dashboard from node_modules and from package-lock.json
-- Still crashes -
More drastically, i have tried to remove everything from the /data folder and redeployed the container.
-- The container then starts, but as soon as i re-download dashboard from the palette manager (it downloads v.2.19.4) then the container is unable to start when stopped via docker. -
I have tried using;
docker run -it --name nodered -p 1880:1880 nodered/node-red:1.0.4-12-arm32v7
-- This works - but obviously doesnt use my filesystem on the host, which leads me to believe theres a problem with one of the files in the /data/nodered/data folder (maybe corruption). But i have cleaned out that folder, started from a fresh nodered container and downloaded dashboard - then it doesnt restart.
So, as you can tell I cannot get to the bottom of this problem, when i try to install node-red-dashboard@2.19.3 manually (I only have a short window to do this before the container is self-destructed and i get thrown out of the bash session)
pi@p1:~ $ docker exec -it <container> /bin/bas
bash-5.0# npm install node-red-dashboard@2.19.3
It comes back with:
bash-5.0# npm install --reinstall node-red-dashboard@2.19.3
npm notice created a lockfile as package-lock.json. You should commit this file.
+ node-red-dashboard@2.19.3
added 47 packages from 42 contributors and audited 1527 packages in 21.641s
13 packages are looking for funding
run `npm fund` for details
found 9 low severity vulnerabilities
run `npm audit fix` to fix them, or `npm audit` for details
bash-5.0#
But then when i stop the container and restart i get the same thing and it doesn't start up.
Are there any other ways to debug the Nodered startup??
I am now losing the plot and cannot get my head from unintentionally banging itself on the desk...
- I don't think its a networking issue - as the container will run in standalone mode (docker run..)
- I don't think its a flows.json problem - as when removed the problem still persists
- I don't think its a docker image problem - as the same problem exhibits over several different images
- I think this could be down to either;
- New dashboard package incompatible with Nodered v1.0.4 or docker
- Something is corrupt inside my node_modules folder or package-lock.json file (cant upload as new user) - but starting with a fresh version of each of these doesnt help...
Any help is much appreciated, and if any more info is needed - let me know an ill post back
Thanks!
