Hi, I'm having a problem that I've been trying to solve for a few days now....but cannot seem to get anywhere with it...I have been using Nodered now for close to a year, and never had problems like i am having now
- Trouble starting a Nodered Docker Container via docker stack deploy
- Started I Think since updating node-red-dashboard to version 2.19.4
- Now unable to start the nodered container at all via docker deploy commands
- Have tried numerous ways to tackle the problem but it still persists
(8x) Cluster of Raspberry Pi 4 -(Hostnames: p1, p2, p3...etc...p7, p8) all running Raspbian Buster Lite
[ Since problems have started I am only deploying on a single node - p1 (docker manager) ]
Running Docker v.19.03.8
Node-red image: nodered/node-red:1.0.4-12-arm32v7
Basically, this setup has been running fine for months (actually using dashboard v2.19.3) until I recently tried to update the dashboard node to v.2.19.4. Now my Nodered container will not start at all...
I am bind mounting the host directory /data/nodered/data to the container volume
The directory </data/nodered/data> contains...
pi@p1:/data/nodered/data $ ls -aln total 90 drwxr-xr-x 5 1000 1000 4096 Mar 30 23:56 . drwxr-xr-x 9 1000 1000 4096 Mar 30 03:15 .. -rw-r--r-- 1 0 0 15284 Mar 30 23:56 .config.json -rw-r--r-- 1 0 0 15285 Mar 30 22:46 .config.json.backup -rw-r--r-- 1 0 0 483 Mar 30 22:19 flows.json drwxr-xr-x 3 0 0 4096 Mar 30 21:26 lib drwxr-xr-x 70 1000 1000 4096 Mar 30 22:44 node_modules -rw-r--r-- 1 0 0 232 Mar 30 22:44 package.json -rw-r--r-- 1 1000 1000 26776 Mar 30 22:45 package-lock.json -rw-r--r-- 1 0 0 12478 Mar 30 21:26 settings.js drwxr-xr-x 2 1000 1000 4096 Mar 30 22:07 tmp
^ Sorry if the formatting messes up here, i will try and fix it if it does..
My compose.yml file for docker stack deploy.....contains;
services: nodered: image: nodered/node-red:1.0.3-4-12-arm32v7 # image: nodered/node-red:1.0.3-4-12-minimal-arm32v7 # image: nodered/node-red:1.0.4-12-arm32v7 # image: nodered/node-red:1.0.4-12-minimal-arm32v7 # entrypoint: "npm start -- --userDir /data --safe" ports: - "1880:1880" user: "0" # user: "1000:1000" environment: TZ: "Europe/London" deploy: mode: replicated replicas: 1 restart_policy: condition: any delay: 25s max_attempts: 5 window: 90s placement: constraints: - node.hostname==p1 volumes: - "/data/nodered/data/:/data/" - "/etc/timezone:/etc/timezone:ro" - "/etc/localtime:/etc/localtime:ro"
The logs i am getting via docker logs -f stack_nodered are not very helpful.
I ahve also set the log level to 'trace' but there's still not much to deliver there
pi@p1:~ $ docker service logs -f stack_nodered stack_nodered.1.hrljun5ah6oh@p1 | stack_nodered.1.hrljun5ah6oh@p1 | > email@example.com start /usr/src/node-red stack_nodered.1.hrljun5ah6oh@p1 | > node $NODE_OPTIONS node_modules/node-red/red.js $FLOWS "--userDir" "/data" stack_nodered.1.hrljun5ah6oh@p1 | stack_nodered.1.hrljun5ah6oh@p1 | 30 Mar 23:25:06 - [info] stack_nodered.1.hrljun5ah6oh@p1 | stack_nodered.1.hrljun5ah6oh@p1 | Welcome to Node-RED stack_nodered.1.hrljun5ah6oh@p1 | =================== stack_nodered.1.hrljun5ah6oh@p1 | stack_nodered.1.hrljun5ah6oh@p1 | 30 Mar 23:25:06 - [info] Node-RED version: v1.0.3 stack_nodered.1.hrljun5ah6oh@p1 | 30 Mar 23:25:06 - [info] Node.js version: v12.15.0 stack_nodered.1.hrljun5ah6oh@p1 | 30 Mar 23:25:06 - [info] Linux 4.19.97-v7l+ arm LE stack_nodered.1.hrljun5ah6oh@p1 | 30 Mar 23:25:06 - [info] Loading palette nodes stack_nodered.1.xcygutoh0ga2@p1 | stack_nodered.1.xcygutoh0ga2@p1 | > firstname.lastname@example.org start /usr/src/node-red stack_nodered.1.xcygutoh0ga2@p1 | > node $NODE_OPTIONS node_modules/node-red/red.js $FLOWS "--userDir" "/data" stack_nodered.1.xcygutoh0ga2@p1 | stack_nodered.1.xcygutoh0ga2@p1 | 30 Mar 23:27:09 - [info] stack_nodered.1.xcygutoh0ga2@p1 | stack_nodered.1.xcygutoh0ga2@p1 | Welcome to Node-RED stack_nodered.1.xcygutoh0ga2@p1 | =================== stack_nodered.1.xcygutoh0ga2@p1 | stack_nodered.1.xcygutoh0ga2@p1 | 30 Mar 23:27:09 - [info] Node-RED version: v1.0.3 stack_nodered.1.xcygutoh0ga2@p1 | 30 Mar 23:27:09 - [info] Node.js version: v12.15.0 stack_nodered.1.xcygutoh0ga2@p1 | 30 Mar 23:27:09 - [info] Linux 4.19.97-v7l+ arm LE stack_nodered.1.xcygutoh0ga2@p1 | 30 Mar 23:27:10 - [info] Loading palette nodes
I know the logs show the image for version 1.0.3 - but it's exactly the same for v.1.0.4
The log just repeats and repeats without much useful information.
If i change the settings.js so include trace debugging, then the log shows the palette nodes being loaded - it reaches up to 'node-red-tail' before it bums out and restarts. But i know that node package comes pre-installed with node-red, so that cant be the problem.
Things I have Tried
- I have tried to use the --safe flag to start without the flows running
In docker compose...
entrypoint: "npm start -- --userDir /data --safe"
-- This also doesnt start - telling me my flow.json file is probably OK
I have tried using several images - 1.0.4....1.0.3....1.0.4-minimal......1.0.3-minimal
-- Same thing happens - so it cant be the image version
I have tried $ 'docker exec -it <> /bin/bash' into the container so I can npm install email@example.com manually,
-- When the container is restarted it crashes and never starts.
I removed node-red-dashboard from node_modules and from package-lock.json
-- Still crashes
More drastically, i have tried to remove everything from the /data folder and redeployed the container.
-- The container then starts, but as soon as i re-download dashboard from the palette manager (it downloads v.2.19.4) then the container is unable to start when stopped via docker.
I have tried using;
docker run -it --name nodered -p 1880:1880 nodered/node-red:1.0.4-12-arm32v7
-- This works - but obviously doesnt use my filesystem on the host, which leads me to believe theres a problem with one of the files in the /data/nodered/data folder (maybe corruption). But i have cleaned out that folder, started from a fresh nodered container and downloaded dashboard - then it doesnt restart.
So, as you can tell I cannot get to the bottom of this problem, when i try to install firstname.lastname@example.org manually (I only have a short window to do this before the container is self-destructed and i get thrown out of the bash session)
pi@p1:~ $ docker exec -it <container> /bin/bas bash-5.0# npm install email@example.com
It comes back with:
bash-5.0# npm install --reinstall firstname.lastname@example.org npm notice created a lockfile as package-lock.json. You should commit this file. + email@example.com added 47 packages from 42 contributors and audited 1527 packages in 21.641s 13 packages are looking for funding run `npm fund` for details found 9 low severity vulnerabilities run `npm audit fix` to fix them, or `npm audit` for details bash-5.0#
But then when i stop the container and restart i get the same thing and it doesn't start up.
Are there any other ways to debug the Nodered startup??
I am now losing the plot and cannot get my head from unintentionally banging itself on the desk...
- I don't think its a networking issue - as the container will run in standalone mode (docker run..)
- I don't think its a flows.json problem - as when removed the problem still persists
- I don't think its a docker image problem - as the same problem exhibits over several different images
- I think this could be down to either;
- New dashboard package incompatible with Nodered v1.0.4 or docker
- Something is corrupt inside my node_modules folder or package-lock.json file (cant upload as new user) - but starting with a fresh version of each of these doesnt help...
Any help is much appreciated, and if any more info is needed - let me know an ill post back