Device Agent Nodered Automatically Restarts

Vendors FlowFuse
1

Hello , How do we debug this kind of issues ? The device agent nodered keeps on restarting and it has happened up to 17 times now for today (22 April 2024) , I dont think this behaviour is normal ?

4/22/2024 6:09:19 AM  []        <--- Last few GCs --->
4/22/2024 6:09:19 AM  []        [2540446:0x3e61bd30]   148658 ms: Scavenge 508.6 (519.4) -> 508.0 (519.9) MB, 27.4 / 0.0 ms  (average mu = 0.147, current mu = 0.090) allocation failure 
4/22/2024 6:09:19 AM  []        [2540446:0x3e61bd30]   148687 ms: Scavenge 509.1 (519.9) -> 508.6 (520.9) MB, 17.1 / 0.0 ms  (average mu = 0.147, current mu = 0.090) allocation failure 
4/22/2024 6:09:19 AM  []        <--- JS stacktrace --->
4/22/2024 6:09:19 AM  []        FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
4/22/2024 6:09:19 AM  []         1: 0xb01764 node::Abort() [node-red]
4/22/2024 6:09:19 AM  []         2: 0xa22604 void node::FPrintF<>(_IO_FILE*, char const*) [node-red]
4/22/2024 6:09:19 AM  []         3: 0xcd0b18 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node-red]
4/22/2024 6:09:19 AM  []         4: 0xcd0ce8 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node-red]
4/22/2024 6:09:19 AM  []         5: 0xe8ca04  [node-red]
4/22/2024 6:09:19 AM  []         6: 0xe8d48c  [node-red]
4/22/2024 6:09:19 AM  []         7: 0xe9b8d4  [node-red]
4/22/2024 6:09:19 AM  []         8: 0xe9c0f4 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node-red]
4/22/2024 6:09:19 AM  []         9: 0xe9ee08 v8::internal::Heap::HandleGCRequest() [node-red]
4/22/2024 6:09:19 AM  []        10: 0xe2f53c v8::internal::StackGuard::HandleInterrupts() [node-red]
4/22/2024 6:09:19 AM  []        11: 0x11c5810 v8::internal::Runtime_StackGuard(int, unsigned long*, v8::internal::Isolate*) [node-red]
4/22/2024 6:09:19 AM  []        12: 0x15874ec  [node-red]
4/22/2024 6:09:19 AM  [info]    Node-RED stopped unexpectedly - restarting
4/22/2024 6:09:19 AM  [info]    Starting Node-RED
4/22/2024 6:09:21 AM  [info]    Welcome to Node-RED

2nd Time It Happened

4/22/2024 6:12:26 AM  []        <--- Last few GCs --->
4/22/2024 6:12:26 AM  []        [2540954:0x33065d30]   184669 ms: Scavenge 508.9 (518.6) -> 508.7 (521.4) MB, 10.1 / 0.0 ms  (average mu = 0.129, current mu = 0.089) allocation failure 
4/22/2024 6:12:26 AM  []        <--- JS stacktrace --->
4/22/2024 6:12:26 AM  []        FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
4/22/2024 6:12:26 AM  []         1: 0xb01764 node::Abort() [node-red]
4/22/2024 6:12:26 AM  []         2: 0xa22604 void node::FPrintF<>(_IO_FILE*, char const*) [node-red]
4/22/2024 6:12:26 AM  []         3: 0xcd0b18 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node-red]
4/22/2024 6:12:26 AM  []         4: 0xcd0ce8 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node-red]
4/22/2024 6:12:26 AM  []         5: 0xe8ca04  [node-red]
4/22/2024 6:12:26 AM  []         6: 0xe8d48c  [node-red]
4/22/2024 6:12:26 AM  []         7: 0xe9b8d4  [node-red]
4/22/2024 6:12:26 AM  []         8: 0xe9c0f4 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node-red]
4/22/2024 6:12:26 AM  []         9: 0xe9ee08 v8::internal::Heap::HandleGCRequest() [node-red]
4/22/2024 6:12:26 AM  []        10: 0xe2f53c v8::internal::StackGuard::HandleInterrupts() [node-red]
4/22/2024 6:12:26 AM  []        11: 0x11c5810 v8::internal::Runtime_StackGuard(int, unsigned long*, v8::internal::Isolate*) [node-red]
4/22/2024 6:12:26 AM  []        12: 0x15874ec  [node-red]
4/22/2024 6:12:26 AM  [info]    Node-RED stopped unexpectedly - restarting
4/22/2024 6:12:26 AM  [info]    Starting Node-RED
4/22/2024 6:12:28 AM  [info]    Welcome to Node-RED

3rd Time it happened

4/22/2024 6:15:02 AM  []        <--- Last few GCs --->
4/22/2024 6:15:02 AM  []        [2541534:0x2040cd30]   153550 ms: Scavenge 507.9 (519.3) -> 507.3 (519.8) MB, 15.5 / 0.0 ms  (average mu = 0.149, current mu = 0.097) allocation failure 
4/22/2024 6:15:02 AM  []        [2541534:0x2040cd30]   153575 ms: Scavenge 508.4 (519.8) -> 507.9 (520.5) MB, 14.5 / 0.0 ms  (average mu = 0.149, current mu = 0.097) allocation failure 
4/22/2024 6:15:02 AM  []        <--- JS stacktrace --->
4/22/2024 6:15:02 AM  []        FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
4/22/2024 6:15:02 AM  []         1: 0xb01764 node::Abort() [node-red]
4/22/2024 6:15:02 AM  []         2: 0xa22604 void node::FPrintF<>(_IO_FILE*, char const*) [node-red]
4/22/2024 6:15:02 AM  []         3: 0xcd0b18 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node-red]
4/22/2024 6:15:02 AM  []         4: 0xcd0ce8 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node-red]
4/22/2024 6:15:02 AM  []         5: 0xe8ca04  [node-red]
4/22/2024 6:15:02 AM  []         6: 0xe8d48c  [node-red]
4/22/2024 6:15:02 AM  []         7: 0xe9b8d4  [node-red]
4/22/2024 6:15:02 AM  []         8: 0xe9c0f4 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node-red]
4/22/2024 6:15:02 AM  []         9: 0xe9ee08 v8::internal::Heap::HandleGCRequest() [node-red]
4/22/2024 6:15:02 AM  []        10: 0xe2f53c v8::internal::StackGuard::HandleInterrupts() [node-red]
4/22/2024 6:15:02 AM  []        11: 0x11c5810 v8::internal::Runtime_StackGuard(int, unsigned long*, v8::internal::Isolate*) [node-red]
4/22/2024 6:15:02 AM  []        12: 0x15874ec  [node-red]
4/22/2024 6:15:02 AM  [info]    Node-RED stopped unexpectedly - restarting
4/22/2024 6:15:02 AM  [info]    Starting Node-RED
4/22/2024 6:15:04 AM  [info]    Welcome to Node-RED

4th Time It Happened

4/22/2024 6:19:50 AM  []        <--- Last few GCs --->
4/22/2024 6:19:50 AM  []        [2542004:0x5fecd30]   285294 ms: Scavenge 507.8 (518.9) -> 507.2 (519.1) MB, 14.5 / 0.0 ms  (average mu = 0.134, current mu = 0.091) allocation failure 
4/22/2024 6:19:50 AM  []        [2542004:0x5fecd30]   285318 ms: Scavenge 508.4 (519.1) -> 507.8 (519.9) MB, 14.1 / 0.0 ms  (average mu = 0.134, current mu = 0.091) allocation failure 
4/22/2024 6:19:50 AM  []        <--- JS stacktrace --->
4/22/2024 6:19:50 AM  []        FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
4/22/2024 6:19:50 AM  []         1: 0xb01764 node::Abort() [node-red]
4/22/2024 6:19:50 AM  []         2: 0xa22604 void node::FPrintF<>(_IO_FILE*, char const*) [node-red]
4/22/2024 6:19:50 AM  []         3: 0xcd0b18 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node-red]
4/22/2024 6:19:50 AM  []         4: 0xcd0ce8 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node-red]
4/22/2024 6:19:50 AM  []         5: 0xe8ca04  [node-red]
4/22/2024 6:19:50 AM  []         6: 0xe8d48c  [node-red]
4/22/2024 6:19:50 AM  []         7: 0xe9b8d4  [node-red]
4/22/2024 6:19:50 AM  []         8: 0xe9c0f4 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node-red]
4/22/2024 6:19:50 AM  []         9: 0xe9ee08 v8::internal::Heap::HandleGCRequest() [node-red]
4/22/2024 6:19:50 AM  []        10: 0xe2f53c v8::internal::StackGuard::HandleInterrupts() [node-red]
4/22/2024 6:19:50 AM  []        11: 0x11c5810 v8::internal::Runtime_StackGuard(int, unsigned long*, v8::internal::Isolate*) [node-red]
4/22/2024 6:19:50 AM  []        12: 0x15874ec  [node-red]
4/22/2024 6:19:50 AM  [info]    Node-RED stopped unexpectedly - restarting
4/22/2024 6:19:50 AM  [info]    Starting Node-RED
4/22/2024 6:19:52 AM  [info]    Welcome to Node-RED

4th Time It Happened

4/22/2024 6:24:24 AM  []        <--- Last few GCs --->
4/22/2024 6:24:24 AM  []        [2542877:0x8d62d30]   272079 ms: Scavenge 510.0 (520.9) -> 509.4 (521.4) MB, 13.8 / 0.0 ms  (average mu = 0.136, current mu = 0.100) allocation failure 
4/22/2024 6:24:24 AM  []        <--- JS stacktrace --->
4/22/2024 6:24:24 AM  []        FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
4/22/2024 6:24:24 AM  []         1: 0xb01764 node::Abort() [node-red]
4/22/2024 6:24:24 AM  []         2: 0xa22604 void node::FPrintF<>(_IO_FILE*, char const*) [node-red]
4/22/2024 6:24:24 AM  []         3: 0xcd0b18 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node-red]
4/22/2024 6:24:24 AM  []         4: 0xcd0ce8 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node-red]
4/22/2024 6:24:24 AM  []         5: 0xe8ca04  [node-red]
4/22/2024 6:24:24 AM  []         6: 0xe8d48c  [node-red]
4/22/2024 6:24:24 AM  []         7: 0xe9b8d4  [node-red]
4/22/2024 6:24:24 AM  []         8: 0xe9c0f4 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node-red]
4/22/2024 6:24:24 AM  []         9: 0xe9ee08 v8::internal::Heap::HandleGCRequest() [node-red]
4/22/2024 6:24:24 AM  []        10: 0xe2f53c v8::internal::StackGuard::HandleInterrupts() [node-red]
4/22/2024 6:24:24 AM  []        11: 0x11c5810 v8::internal::Runtime_StackGuard(int, unsigned long*, v8::internal::Isolate*) [node-red]
4/22/2024 6:24:24 AM  []        12: 0x15874ec  [node-red]
4/22/2024 6:24:24 AM  [info]    Node-RED stopped unexpectedly - restarting
4/22/2024 6:24:24 AM  [info]    Starting Node-RED
4/22/2024 6:24:26 AM  [info]    Welcome to Node-RED
2

Hi, did you notice the out of memory warning on each of these?

The error get thrown in a variety of situations, here are a few:

  • When attempting to process large amounts of data or execute complex algorithms that require a lot of memory.
  • When running a Node.js server with a large number of connections or concurrent requests.
  • When running a Node.js application in a container or virtual machine with limited memory resources.

Also note, the issue is with the node-red process not the device agent:

What that means is this would happen running node-red standalone - it is most likely an issue with what your flows are doing or a contrib node you are using.

What kind of work is node-red doing?
What OS and hardware is it running on?
How much memory is available to node-red?
Can you list it the contrib nodes installed?

3
  1. Receive data from chirpstack through MQTT (Localhost connections)
  2. its reading 10 latest rows from postgres every 30 seconds
  3. Send the data to google cloud pubsub
  4. Update the table columns once the data has been published to pubsub

Basically what it does is basically syncing the local db from the remote locations with the cloud db located on the cloud

The OS is rakpiOS which is based on raspbian/debian and the hardware is CM4 8gb with SSD of 256 Gb

How do i check the memory allocation to nodered ? What commands should i run ?

Contrib Nodes that i have installed are

  1. @chirpstack/node-red-contrib-chirpstack
  2. @digitaloak/node-red-contrib-digitaloak-postgresql
  3. @flowfuse/node-red-dashboard
  4. @flowfuse/nr-file-nodes
  5. @flowfuse/nr-project-nodes
  6. @gregoriusrippenstein/node-red-contrib-flow2uml
  7. node-red
  8. node-red-contrib-chronos
  9. node-red-contrib-file-function-ext
  10. node-red-contrib-isonline
  11. node-red-contrib-loop
  12. node-red-contrib-loop-processing
  13. node-red-contrib-simple-gate
  14. node-red-contrib-simple-queue
  15. node-red-node-sqlite

Do note that this issue only happens on this device , all the other device that are running the same set of flows are working propoerly