What is causing my NR to crash/restart?

General
1

Hi all,

I'm running NR 1.0.3 in Docker. Every once in a while NR seems to crash. Sometimes after a week or so, sometimes even couple of times per day. How can I find out what causes this? In the logs (that I view via Portainer) I do see that NR is starting again (I think docker-compose triggers that via restart:always).

But there are no error messages visible indicating that NR crashed right before the startup-related messages.
F.i. I see:

13 Dec 19:04:47 - [info] Starting modified nodes
13 Dec 19:04:47 - [info] Started modified nodes

> node-red-docker@1.0.3 start /usr/src/node-red
> node $NODE_OPTIONS node_modules/node-red/red.js $FLOWS "--userDir" "/data"
13 Dec 19:17:52 - [info] 
Welcome to Node-RED
===================
13 Dec 19:17:52 - [info] Node-RED version: v1.0.3
13 Dec 19:17:52 - [info] Node.js  version: v10.17.0
13 Dec 19:17:52 - [info] Linux 4.19.75-v7+ arm LE
13 Dec 19:17:54 - [info] Loading palette nodes

How can I find out what causes NR to appearently crash?

By the way, I have several other docker images running on the Pi and they are all stable. So I don't think it has anything to do with the OS or hardware.

2

sorry - not without any more information than that.

3

Have a look in /var/log/syslog and see if there are any clues at the point of restart.

4

I see the following there:

Dec 13 19:17:40 nas systemd[1]: run-docker-runtime\x2drunc-moby-12f04cf0d5768b146b006e78f0d7d677d178473d9274816843b4ad7ec776f975-runc.Hq6h50.mount: Succeeded.
Dec 13 19:17:41 nas containerd[471]: time="2019-12-13T19:17:41.191912854+01:00" level=info msg="shim reaped" id=12f04cf0d5768b146b006e78f0d7d677d178473d9274816843b4ad7ec776f975
Dec 13 19:17:41 nas containerd[471]: time="2019-12-13T19:17:41.192412018+01:00" level=warning msg="cleaning up after killed shim" id=12f04cf0d5768b146b006e78f0d7d677d178473d9274816843b4ad7ec776f975 namespace=moby
Dec 13 19:17:41 nas dockerd[473]: time="2019-12-13T19:17:41.202586814+01:00" level=warning msg="failed to delete process" container=12f04cf0d5768b146b006e78f0d7d677d178473d9274816843b4ad7ec776f975 error="not found" module=libcontainerd namespace=moby process=310b9e1317842494223784ed82ec742a20b6ade3259808667cd669270e2f75f7
Dec 13 19:17:41 nas dockerd[473]: time="2019-12-13T19:17:41.458263205+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

And afterwards lots of messages that I believe are related to Docker restarting the container, so I left them out.

5

It would seem to be related to running out of memory (maybe). See this discussion for similar https://github.com/containerd/containerd/issues/2202

6

Thanks for thinking along. Weird that it only happens to the NR container.
I'll keep monitoring the NR container memory usage via Portainer for some time.
At this moment it is around 100MB.