Machine times between reboots/lockups reducing

Here's the thing:

On one of the machines I get these errors now and then.

vcgencmd, ping and exec nodes with spawn errors.

I'll try to capture one when it happens.
Though I have posted them before, they were.... dismissed as non-important or trivial.

But that may be a whole other issue.

Thanks @dceejay .

I saw / noticed that and was a bit worried.

Yes, that is a possibility.

That machine is the oldest and oldest (and oldest) RasPi.

Age, version and running.

It is an early one with only 2 USB ports and limited GPIO pins.
It also has a RTC (I2C) connected and is my NTP server.

On power:
A thought:

I have a couple of those USB battery things.
Their input is the same as the RasPi and their output is the bigger USB port.
So I could plug the cable power the RasPi into the battery then get a cable with the correct connectors between the battery and Pi.

That way any spikes would be absorbed by the battery and any power drops would be filled in by the batter.

Does that sound like a possibility worth trying?
Of course if there is constant over voltage output from the supply that's a whole different story, but none of the pies seem to be getting / suffering from that from what I have read.

Most of the battery packs (cheaper ones) will not charge and discharge at the same time -try it and see what happens

Craig

1 Like

Ok thanks.

I just put it out there as a possibility.

Just for the sake of it:

Could I replace the smooth node and put a RBE node there.

That way only if it changes does the message get through.

It would but you need still a couple of things. Firstly cables are a common issue so that USB tester is still your best friend.

Secondly, as Craig says, most of the USB powerbanks can't deliver power and be on charge at the same time. If you have one that does, use the tester on the Pi end to make sure that it is getting enough voltage and current.

Mustool ud18 usb3.0/dc/type-c 18 in 1 usb tester app dc digital voltmeter ammeter hd color screen 6-bit high precision display ud18 bluetooth digital meter Sale - Banggood.com

Yes, I think that is a NO to the question.

The reboots don't happen at the same times on all machines.

Sorry for the delay in the reply.

A lot of stuff to digest.

Just a small update on things:

I bought/built a small USB voltage tester.

This one

Nothing too fancy, but it seems to work.

The Batteries and using them while charging.

Ok, got the kit plugged it into a 240v / USB thingy. Green LED glows. GOOD.

Plug it into the battery. GREEN. GOOD.

Plug in charger.

The YELLOW LED is blinking.

Reading the docs, that is LOW VOLTAGE.

Maybe because it is also charging the battery - slightly off 100% charged.

But it does supply power while charging.

More research needed though before I try it out.

These early ones are/were well known for being very sensitive to power fluctuations... On mine just plugging in a tiny wireless keyboard usb adapter (while powered on) is enough to cause it to restart.

2 Likes

Sure, it will make a bit of a difference....You will get a lot more throughput on it though.... The # Sockets changes quite a bit on normal running(on my system anyway), so the RBE node will pass a lot through... The smooth node just removes the spikes for me and smoothes out the graph quite a bit, makes it easier to spot the trend.... I am not so much interested in the # of sockets open as to whether the number is climbing or falling over a period....

Cya

E

Looking at what I am getting, it is a wave

Part of a log from one machine:

2021-06-03 10:57:55 --> 260
2021-06-03 10:59:53 --> 258
2021-06-03 11:05:53 --> 259
2021-06-03 11:17:53 --> 258
2021-06-03 11:19:53 --> 259
2021-06-03 11:29:53 --> 255
2021-06-03 11:39:59 --> 292
2021-06-03 11:41:56 --> 255
2021-06-03 13:11:56 --> 256
2021-06-03 13:17:58 --> 261
2021-06-03 13:19:56 --> 258
2021-06-03 13:36:06 --> 294
2021-06-03 13:38:00 --> 257
2021-06-03 13:42:00 --> 258
2021-06-03 13:44:00 --> 259
2021-06-03 14:12:00 --> 258
2021-06-03 14:16:01 --> 259
2021-06-03 14:18:01 --> 258
2021-06-03 14:22:01 --> 259
2021-06-03 14:24:03 --> 264
2021-06-03 14:26:01 --> 260
2021-06-03 16:18:04 --> 264
2021-06-03 16:20:04 --> 260
2021-06-03 16:32:05 --> 266
2021-06-03 16:34:05 --> 273
2021-06-03 16:36:05 --> 267
2021-06-03 16:38:05 --> 273
2021-06-03 16:40:05 --> 282
2021-06-03 16:42:05 --> 281
2021-06-03 16:44:05 --> 299
2021-06-03 16:46:05 --> 293
2021-06-03 16:48:05 --> 299
2021-06-03 16:50:06 --> 293
2021-06-03 16:52:06 --> 287
2021-06-03 16:54:06 --> 306
2021-06-03 16:56:11 --> 300
2021-06-03 16:58:16 --> 260
2021-06-03 17:00:13 --> 314
2021-06-03 17:02:10 --> 256
2021-06-03 17:24:10 --> 257
2021-06-03 17:52:16 --> 291
2021-06-03 17:54:13 --> 257
2021-06-03 20:10:13 --> 258
2021-06-03 20:18:14 --> 265
2021-06-03 20:22:14 --> 259
2021-06-03 20:26:15 --> 263
2021-06-03 20:28:14 --> 259
2021-06-03 20:30:14 --> 271
2021-06-03 20:32:14 --> 265
2021-06-03 20:36:15 --> 270
2021-06-03 20:38:15 --> 272
2021-06-03 20:40:15 --> 259
2021-06-03 20:46:15 --> 263
2021-06-03 20:48:15 --> 259
2021-06-03 20:50:15 --> 272
2021-06-03 20:52:15 --> 279
2021-06-03 20:54:15 --> 273
2021-06-03 20:56:15 --> 270
2021-06-03 20:58:15 --> 272
2021-06-03 21:00:15 --> 273
2021-06-03 21:02:15 --> 285
2021-06-03 21:04:16 --> 280
2021-06-03 21:08:15 --> 285
2021-06-03 21:12:16 --> 297
2021-06-03 21:14:16 --> 285
2021-06-03 21:16:16 --> 297
2021-06-03 21:18:16 --> 304
2021-06-03 21:20:16 --> 278
2021-06-03 21:22:16 --> 275
2021-06-03 21:24:16 --> 277
2021-06-03 21:26:16 --> 293
2021-06-03 21:28:16 --> 278
2021-06-03 21:32:16 --> 282
2021-06-03 21:34:17 --> 294
2021-06-03 21:36:16 --> 284
2021-06-03 21:38:16 --> 289
2021-06-03 21:40:16 --> 284
2021-06-03 21:42:17 --> 288
2021-06-03 21:44:27 --> 256
2021-06-03 21:56:18 --> 262
2021-06-03 21:58:17 --> 256
2021-06-03 22:00:17 --> 262
2021-06-03 22:04:17 --> 275
2021-06-03 22:06:17 --> 269
2021-06-03 22:10:18 --> 270
2021-06-03 22:12:18 --> 269
2021-06-03 22:14:18 --> 270
2021-06-03 22:18:18 --> 269
2021-06-03 22:20:18 --> 281
2021-06-03 22:22:18 --> 283
2021-06-03 22:24:18 --> 284
2021-06-03 22:26:18 --> 283
2021-06-03 22:28:18 --> 290
2021-06-03 22:30:18 --> 291
2021-06-03 22:34:18 --> 303
2021-06-03 22:36:29 --> 256
2021-06-03 22:38:27 --> 293
2021-06-03 22:40:24 --> 256
2021-06-04 01:06:24 --> 262
2021-06-04 01:08:25 --> 269
2021-06-04 01:10:25 --> 276
2021-06-04 01:12:25 --> 277
2021-06-04 01:14:25 --> 276
2021-06-04 01:18:25 --> 281

Hi Andrew,

Yeh...it is in a bit of a wave by the looks of it, but the highs don't seem to be getting higher per the 1/2 day or so cycle sample, smoothing it to about 10 readings would be easier to see the trend... My gut feel is that this is not the problem area, unless, of course, something sparks off a problem that this particular sample doesn't cover.... In my case it was a node that was leaving things dangling....

Cheers

E

I don't know if this is (or not) relevant to the problem.

Just now I got these errors on one of the machines.

11:49:08

{"message":"Error: spawn ENOMEM","source":{"id":"bd376285.3f55e8","type":"exec","count":1},"stack":"Error: spawn ENOMEM\n    at ChildProcess.spawn (internal/child_process.js:408:11)\n    at spawn (child_process.js:553:9)\n    at Object.execFile (child_process.js:237:17)\n    at exec (child_process.js:158:25)\n    at ExecNode._inputCallback (/usr/lib/node_modules/node-red/node_modules/@node-red/nodes/core/function/90-exec.js:134:29)\n    at /usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:203:26\n    at Object.trigger (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/hooks.js:113:9)\n    at ExecNode.Node._emitInput (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:195:11)\n    at ExecNode.Node.emit (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:179:25)\n    at ExecNode.Node.receive (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:476:10)\n    at Immediate._onImmediate (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/flows/Flow.js:657:5..."}

11:49:08

{"message":"Error: spawn ENOMEM","source":{"id":"d1173d22.231328","type":"exec","name":"Check Temp","count":1},"stack":"Error: spawn ENOMEM\n    at ChildProcess.spawn (internal/child_process.js:408:11)\n    at spawn (child_process.js:553:9)\n    at Object.execFile (child_process.js:237:17)\n    at exec (child_process.js:158:25)\n    at ExecNode._inputCallback (/usr/lib/node_modules/node-red/node_modules/@node-red/nodes/core/function/90-exec.js:134:29)\n    at /usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:203:26\n    at Object.trigger (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/hooks.js:113:9)\n    at ExecNode.Node._emitInput (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:195:11)\n    at ExecNode.Node.emit (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:179:25)\n    at ExecNode.Node.receive (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:476:10)\n    at Immediate._onImmediate (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/flows/Flow.js:657:5..."}

11:49:08

{"message":"Error: spawn ENOMEM","source":{"id":"bd376285.3f55e8","type":"exec","count":1},"stack":"Error: spawn ENOMEM\n    at ChildProcess.spawn (internal/child_process.js:408:11)\n    at spawn (child_process.js:553:9)\n    at Object.execFile (child_process.js:237:17)\n    at exec (child_process.js:158:25)\n    at ExecNode._inputCallback (/usr/lib/node_modules/node-red/node_modules/@node-red/nodes/core/function/90-exec.js:134:29)\n    at /usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:203:26\n    at Object.trigger (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/hooks.js:113:9)\n    at ExecNode.Node._emitInput (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:195:11)\n    at ExecNode.Node.emit (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:179:25)\n    at ExecNode.Node.receive (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:476:10)\n    at Immediate._onImmediate (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/flows/Flow.js:657:5..."}

11:49:08

{"message":"Error: spawn ENOMEM","source":{"id":"d1173d22.231328","type":"exec","name":"Check Temp","count":1},"stack":"Error: spawn ENOMEM\n    at ChildProcess.spawn (internal/child_process.js:408:11)\n    at spawn (child_process.js:553:9)\n    at Object.execFile (child_process.js:237:17)\n    at exec (child_process.js:158:25)\n    at ExecNode._inputCallback (/usr/lib/node_modules/node-red/node_modules/@node-red/nodes/core/function/90-exec.js:134:29)\n    at /usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:203:26\n    at Object.trigger (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/hooks.js:113:9)\n    at ExecNode.Node._emitInput (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:195:11)\n    at ExecNode.Node.emit (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:179:25)\n    at ExecNode.Node.receive (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:476:10)\n    at Immediate._onImmediate (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/flows/Flow.js:657:5..."}

11:49:17

{"message":"Error: spawn ENOMEM","source":{"id":"3ca1bd56.5515aa","type":"ping","name":"Ping","count":1},"stack":"Error: spawn ENOMEM\n    at ChildProcess.spawn (internal/child_process.js:408:11)\n    at spawn (child_process.js:553:9)\n    at doPing (/home/pi/.node-red/node_modules/node-red-node-ping/88-ping.js:40:14)\n    at PingNode._inputCallback (/home/pi/.node-red/node_modules/node-red-node-ping/88-ping.js:120:36)\n    at /usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:203:26\n    at Object.trigger (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/hooks.js:113:9)\n    at PingNode.Node._emitInput (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:195:11)\n    at PingNode.Node.emit (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:179:25)\n    at PingNode.Node.receive (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:476:10)\n    at Immediate._onImmediate (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/flows/Flow.js:657:52)\n    at processImmediate..."}

11:49:17

{"message":"Error: spawn ENOMEM","source":{"id":"3ca1bd56.5515aa","type":"ping","name":"Ping","count":1},"stack":"Error: spawn ENOMEM\n    at ChildProcess.spawn (internal/child_process.js:408:11)\n    at spawn (child_process.js:553:9)\n    at doPing (/home/pi/.node-red/node_modules/node-red-node-ping/88-ping.js:40:14)\n    at PingNode._inputCallback (/home/pi/.node-red/node_modules/node-red-node-ping/88-ping.js:120:36)\n    at /usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:203:26\n    at Object.trigger (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/hooks.js:113:9)\n    at PingNode.Node._emitInput (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:195:11)\n    at PingNode.Node.emit (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:179:25)\n    at PingNode.Node.receive (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:476:10)\n    at Immediate._onImmediate (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/flows/Flow.js:657:52)\n    at processImmediate..."}

11:50:35

{"message":"Error: spawn ENOMEM","source":{"id":"3ca1bd56.5515aa","type":"ping","name":"Ping","count":1},"stack":"Error: spawn ENOMEM\n    at ChildProcess.spawn (internal/child_process.js:408:11)\n    at spawn (child_process.js:553:9)\n    at doPing (/home/pi/.node-red/node_modules/node-red-node-ping/88-ping.js:40:14)\n    at PingNode._inputCallback (/home/pi/.node-red/node_modules/node-red-node-ping/88-ping.js:120:36)\n    at /usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:203:26\n    at Object.trigger (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/hooks.js:113:9)\n    at PingNode.Node._emitInput (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:195:11)\n    at PingNode.Node.emit (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:179:25)\n    at PingNode.Node.receive (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:476:10)\n    at Immediate._onImmediate (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/flows/Flow.js:657:52)\n    at processImmediate..."}

This is on a RasPi B Rev2.
The OLDEST of them all.

Just got some more:

12:20:26

{"message":"Error: spawn ENOMEM","source":{"id":"3ca1bd56.5515aa","type":"ping","name":"Ping","count":1},"stack":"Error: spawn ENOMEM\n    at ChildProcess.spawn (internal/child_process.js:408:11)\n    at spawn (child_process.js:553:9)\n    at doPing (/home/pi/.node-red/node_modules/node-red-node-ping/88-ping.js:40:14)\n    at PingNode._inputCallback (/home/pi/.node-red/node_modules/node-red-node-ping/88-ping.js:120:36)\n    at /usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:203:26\n    at Object.trigger (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/hooks.js:113:9)\n    at PingNode.Node._emitInput (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:195:11)\n    at PingNode.Node.emit (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:179:25)\n    at PingNode.Node.receive (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:476:10)\n    at Immediate._onImmediate (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/flows/Flow.js:657:52)\n    at processImmediate..."}

12:19:33

{"message":"Error: spawn ENOMEM","source":{"id":"bd376285.3f55e8","type":"exec","count":1},"stack":"Error: spawn ENOMEM\n    at ChildProcess.spawn (internal/child_process.js:408:11)\n    at spawn (child_process.js:553:9)\n    at Object.execFile (child_process.js:237:17)\n    at exec (child_process.js:158:25)\n    at ExecNode._inputCallback (/usr/lib/node_modules/node-red/node_modules/@node-red/nodes/core/function/90-exec.js:134:29)\n    at /usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:203:26\n    at Object.trigger (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/hooks.js:113:9)\n    at ExecNode.Node._emitInput (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:195:11)\n    at ExecNode.Node.emit (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:179:25)\n    at ExecNode.Node.receive (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:476:10)\n    at Immediate._onImmediate (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/flows/Flow.js:657:5..."}

(There is/was a heap of them. I won't post them all. Like 14)

12:23:45

{"message":"Error: spawn ENOMEM","source":{"id":"43d7b6c6.c95118","type":"pythonshell in","name":"wifiscan","count":1},"stack":"Error: spawn ENOMEM\n    at ChildProcess.spawn (internal/child_process.js:408:11)\n    at PythonshellInNode.spawn (child_process.js:553:9)\n    at PythonshellInNode.onInput (/home/pi/.node-red/node_modules/node-red-contrib-pythonshell/src/PythonShellNode.js:63:20)\n    at PythonshellInNode._inputCallback (/home/pi/.node-red/node_modules/node-red-contrib-pythonshell/src/pythonshell.js:35:14)\n    at /usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:203:26\n    at Object.trigger (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/hooks.js:113:9)\n    at PythonshellInNode.Node._emitInput (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:195:11)\n    at PythonshellInNode.Node.emit (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:179:25)\n    at PythonshellInNode.Node.receive (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:476:10)\n    at Immediate._onImmediate ..."}

12:23:45

{"message":"Error: spawn ENOMEM","source":{"id":"78b8e72a.839e7","type":"vcgencmd","count":1},"stack":"Error: spawn ENOMEM\n    at ChildProcess.spawn (internal/child_process.js:408:11)\n    at spawn (child_process.js:553:9)\n    at Object.execFile (child_process.js:237:17)\n    at Object.exec (child_process.js:158:25)\n    at VcGenCmdNode._inputCallback (/home/pi/.node-red/node_modules/node-red-contrib-vcgencmd/vcgencmd.js:106:47)\n    at /usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:203:26\n    at Object.trigger (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/hooks.js:113:9)\n    at VcGenCmdNode.Node._emitInput (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:195:11)\n    at VcGenCmdNode.Node.emit (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:179:25)\n    at VcGenCmdNode.Node.receive (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:476:10)\n    at Immediate._onImmediate (/usr/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/flows/Flow...."}

Oh, other things known:

Uptime 3 weeks 1 day 5 hours.

92% of memory used.

Have you googled for enomem?

ā€œCannot allocate memory.ā€ The system cannot allocate more virtual memory because its capacity is full.

Fairly obvious.

Unless 92% of memory was used already at start 3 weeks, 1day and 5 hours ago, you have something eating memory

What else do you have running on the Pi?

AFAIK I have running:

NTP server

Node-Red

MQTT Broker

Last night I restarted NR on it from the CLI.

After NR started I was told 90% of memory is used.