Nodered container crash everyday

Barbudor · 2 April 2024 10:07

Hello

Since a week I facing a strange crash on one of my nodered container (running in AWS ECS on a FARGATE instance)
There is no memory or CPU overload but something seems to happen that I have not been able to identify
Nodered version is 3.0.2, default docker imagebased on node 16.20.2

You would says that as it happens everyday at the same hour it should be easy to identify the guilty flow but actually I don't have any flow that start at that time, so I'm a bit clue less

Here is what I see in the logs prior to the crash:

2 Apr 09:17:11 - [red] Uncaught Exception:
at processImmediate (node:internal/timers:466:21)
at Immediate.<anonymous> (/usr/src/node-red/node_modules/@node-red/runtime/lib/flows/Flow.js:831:52)
2 Apr 09:17:11 - [error] TypeError: sendEvent.destination.node.receive is not a function
'===================
Welcome to Node-RED
2 Apr 09:18:06 - [info]

I also noticed that just after restart there are those errors that I can't neither link to something:

2 Apr 09:18:49 - [warn]
2 Apr 09:18:49 - [info] Starting flows
TypeError: Cannot read properties of undefined (reading 'id')
at Object.start [as startFlows] (/usr/src/node-red/node_modules/@node-red/runtime/lib/flows/index.js:394:33)
at Flow.start (/usr/src/node-red/node_modules/@node-red/runtime/lib/flows/Flow.js:224:41)
at Subflow.start (/usr/src/node-red/node_modules/@node-red/runtime/lib/flows/Subflow.js:209:62)
at Array.map (<anonymous>)
at /usr/src/node-red/node_modules/@node-red/runtime/lib/flows/Subflow.js:209:95
at Array.map (<anonymous>)
at /usr/src/node-red/node_modules/@node-red/runtime/lib/flows/Subflow.js:209:140
2 Apr 09:19:23 - [info] [mongodb-aleph-in:find in custom_metadata] connecting to mongodb...
at Object.start [as startFlows] (/usr/src/node-red/node_modules/@node-red/runtime/lib/flows/index.js:394:33)
at Flow.start (/usr/src/node-red/node_modules/@node-red/runtime/lib/flows/Flow.js:224:41)
at Subflow.start (/usr/src/node-red/node_modules/@node-red/runtime/lib/flows/Subflow.js:209:62)
at Array.map (<anonymous>)
at /usr/src/node-red/node_modules/@node-red/runtime/lib/flows/Subflow.js:209:95
at Array.map (<anonymous>)
at /usr/src/node-red/node_modules/@node-red/runtime/lib/flows/Subflow.js:209:140
TypeError: Cannot read properties of undefined (reading 'id')
2 Apr 09:19:23 - [info] Started flows
(Use `node --trace-warnings ...` to show where the warning was created)

Any advice to help tracking down where this could happen would be appreciated
Thanks

Colin · 2 April 2024 10:28

Try setting the log level to Trace in settings.js in the hope it will give more information.

Barbudor · 2 April 2024 16:18

Thanks @Colin

I made the change and will restart the container at night
Then we will see if it crashes again tomorrow

Colin · 2 April 2024 20:27

The Trace setting is more about determining what is causing the errors on startup. Though it may give information about the crash too, but less likely.

Barbudor · 4 April 2024 07:58

Ah thanks
However I now have the traces for both

Here are the logs right before the crash

2024-04-04T06:31:37.086Z	4 Apr 06:31:37 - [debug] Flushing localfilesystem context scope 304d6298862b9e4b
2024-04-04T06:34:56.412Z	4 Apr 06:34:56 - [debug] Flushing localfilesystem context scope 304d6298862b9e4b
2024-04-04T06:37:27.891Z	at processImmediate (node:internal/timers:466:21)
2024-04-04T06:37:27.891Z	at Immediate.<anonymous> (/usr/src/node-red/node_modules/@node-red/runtime/lib/flows/Flow.js:831:52)
2024-04-04T06:37:27.891Z	4 Apr 06:37:27 - [error] TypeError: sendEvent.destination.node.receive is not a function
2024-04-04T06:37:27.891Z	4 Apr 06:37:27 - [red] Uncaught Exception:
2024-04-04T06:38:40.550Z	'===================
2024-04-04T06:38:40.550Z	Welcome to Node-RED

I found this Github issue which refers to using node.send()into a promise call back which I'm actually using in a function node:

// Initialize slack
const web = new slackWebApi.WebClient("xxxxxxxxxxxxxxxxxxxxxx");

node.warn(`Slack upload with ${JSON.stringify(msg.payload)}`);

(async () => {
    const result = await web.files.uploadV2(msg.payload);
    msg.payload = result
    node.send(msg)
})();

If this is to be avoided, could you please advise how I should code that instead ?

And the logs during boot issue are those:

2024-04-04T06:39:45.286Z	4 Apr 06:39:45 - [trace] [subflow:be049e0bbf73290d] ------------------|--------------|-----------------
2024-04-04T06:39:45.286Z	4 Apr 06:39:45 - [trace] [subflow:be049e0bbf73290d] start subflow [cdb37cb8bad2f69e/e7ee7117246f4ae3/cab9972dcf655114/09507900c4b343fe]
2024-04-04T06:39:45.286Z	4 Apr 06:39:45 - [trace] [subflow:1cc39ec7a53d287e] start subflow [cdb37cb8bad2f69e/e7ee7117246f4ae3/cab9972dcf655114]
2024-04-04T06:39:45.286Z	4 Apr 06:39:45 - [trace] [subflow:e7ee7117246f4ae3] start subflow [cdb37cb8bad2f69e/e7ee7117246f4ae3]
2024-04-04T06:39:45.288Z	at Object.start [as startFlows] (/usr/src/node-red/node_modules/@node-red/runtime/lib/flows/index.js:394:33)
2024-04-04T06:39:45.288Z	at Flow.start (/usr/src/node-red/node_modules/@node-red/runtime/lib/flows/Flow.js:224:41)
2024-04-04T06:39:45.288Z	at Subflow.start (/usr/src/node-red/node_modules/@node-red/runtime/lib/flows/Subflow.js:209:62)
2024-04-04T06:39:45.288Z	at Array.map (<anonymous>)
2024-04-04T06:39:45.288Z	at /usr/src/node-red/node_modules/@node-red/runtime/lib/flows/Subflow.js:209:95
2024-04-04T06:39:45.288Z	at Array.map (<anonymous>)
2024-04-04T06:39:45.288Z	at /usr/src/node-red/node_modules/@node-red/runtime/lib/flows/Subflow.js:209:140
2024-04-04T06:39:45.288Z	TypeError: Cannot read properties of undefined (reading 'id')
2024-04-04T06:39:45.302Z	4 Apr 06:39:45 - [debug] [subflow:2f9f934b3ee78deb] not starting disabled node : 6388487f96043b52
2024-04-04T06:39:45.302Z	4 Apr 06:39:45 - [trace] [subflow:2f9f934b3ee78deb] start subflow [1ee34aef2976ed38/2f9f934b3ee78deb]
2024-04-04T06:39:45.303Z	4 Apr 06:39:45 - [trace] [subflow:2f9f934b3ee78deb] ------------------|--------------|-----------------

Which seems to locate the problem in subflow e7ee7117246f4ae3
The problem seems to occur in that line (node-red3.0.2):

github.com

node-red/node-red/blob/5365786386e21df74b339a399e854ed89af6394f/packages/node_modules/@node-red/runtime/lib/flows/Subflow.js#L209


      
          
          var subflowInstanceConfig = {
              id: this.subflowInstance.id,
              type: this.subflowInstance.type,
              z: this.subflowInstance.z,
              name: this.subflowInstance.name,
              wires: [],
              _flow: this
          }
          if (this.subflowDef.in) {
              subflowInstanceConfig.wires = this.subflowDef.in.map(function(n) { return n.wires.map(function(w) { return self.node_map[w.id].id;})})
              subflowInstanceConfig._originalWires = clone(subflowInstanceConfig.wires);
          }
          
          this.node = new Node(subflowInstanceConfig);
          this.node.on("input", function(msg) { this.send(msg);});
          // Called when the subflow instance node is being stopped
          this.node.on("close", function(done) {
              this.status({});
              // Stop the complete subflow
              self.stop().finally(done)

One of the 2 .id seems to be non existant

Any idea what could cause that ?
We have other node-red instances running in similar conditions (other AWS regions) with same nodes and subflows and I haven't cound the same log in the last 30 days.

Thanks for any hints on those 2 issues

Barbudor · 10 April 2024 07:45

Hello

Anyone with some hints on those 2 errors ?
The most critical one being the daily crash

Thanks

Jean-Michel

Steve-Mcl · 10 April 2024 09:11

Barbudor:

using node.send()into a promise call back which I'm actually using in a function node:
// Initialize slack
const web = new slackWebApi.WebClient("xxxxxxxxxxxxxxxxxxxxxx");

node.warn(`Slack upload with ${JSON.stringify(msg.payload)}`);

(async () => {
  const result = await web.files.uploadV2(msg.payload);
  msg.payload = result
  node.send(msg)
})();
If this is to be avoided, could you please advise how I should code that instead ?

The function node is top level async await so this is unnecessary.

Something like this would be better (added try-catch to show you options available to you in your function nodes)

// Initialize slack
const web = new slackWebApi.WebClient("xxxxxxxxxxxxxxxxxxxxxx");
node.warn(`Slack upload with ${JSON.stringify(msg.payload)}`);
try {
    const result = await web.files.uploadV2(msg.payload);
    msg.payload = result
    node.send(msg)
} catch (e) {
    node.error(e, msg)
}

FWIW: your original code did not have any error handling inside the IIFE, so it would definitely crash node-red if it failed.

Barbudor · 10 April 2024 10:03

Hi @Steve-Mcl

Many thanks. Still a lot to learn
Indeed adding error handling code is better

I'll change the code as per your advice and let you know within a couple of days

Best regards

Barbudor · 10 April 2024 10:06

Hi @Steve-Mcl

Wouldn't it be required to add a node.done() ?

try {
    const result = await web.files.uploadV2(msg.payload);
    msg.payload = result
    node.send(msg)
    node.done();
} catch (e) {
    node.error(e, msg)
}

Steve-Mcl · 10 April 2024 10:15

Not really necessary unless you are relying on the complete node.

If node.done() is excluded, it will still (implicitly) trigger a complete node - however it will be triggered regardless of success (i.e. even node.error is called / node.send not called) when the function node ends cleanly. By adding the explicit node.done, you control when the complete node is triggered.

In short, node.send will send regardless of the inclusion or exclusion of node.done

Barbudor · 10 April 2024 10:20

Ok, thanks

Barbudor · 11 April 2024 17:11

Hi @Steve-Mcl @Colin

I did 2 changes and the crash didn't occurred this morning

Remove the async and added the try/catch as suggested by Steve.
There was a flow running daily (from one of my colleague) with a function node doing:

msg.payload = msg.entry.leagues_list;
node.send(msg);
node.done();

msg.payload = msg.entry.teams_list;
node.send(msg);
node.done();

return;

I stopped that flow.
I believe that code is not the best, and may be those 2 x node.done() is bad and more likely responsible for the sendEvent.destination.node.receive is not a function error message.
I believe it should be enough to write it like:

msg.payload = msg.entry.leagues_list;
node.send(msg);
msg.payload = msg.entry.teams_list;
return msg;

Do you aggree ?

Thanks

Steve-Mcl · 11 April 2024 17:20

Why do you even need to send 2 messages? The data you want is already present in the msg. Where are you sending these 2 messages that you generate in the function?

as for "better"? No, what you are doing is better suited to a change nodes and wires (that is after all the point of visual programming)

the wires branching here ^ will cause the msg to be duplicated, then the change nodes will set your payloads - but i re-iterate, since your values are already in one node, why not just use them?

Barbudor · 11 April 2024 19:49

Thanks @Steve-Mcl
The visual approach with the 2 changes nodes is definitively the nicest solution

The reason is that there then a split node which create as many messages as elements in both lists.

Another alternative would be to merge both lists into payload and issue a single message to the split node.

"not my flow", I likely would have designed the flow like that anyhow

Thanks again

Topic		Replies	Views
[BUG] subflow - node-red crash Core Development	2	334	12 November 2021
Node-red crashing sendEvent.destination.node.receive is not a function General	13	754	22 October 2022
Nodered crashing General docker	3	100	30 April 2024
Node-RED crashes randomly General docker	7	315	30 October 2023
Node-red crashes General	3	1250	3 July 2019

Nodered container crash everyday

Related Topics