Nodered container crash everyday

Hello

Since a week I facing a strange crash on one of my nodered container (running in AWS ECS on a FARGATE instance)
There is no memory or CPU overload but something seems to happen that I have not been able to identify
Nodered version is 3.0.2, default docker imagebased on node 16.20.2

You would says that as it happens everyday at the same hour it should be easy to identify the guilty flow but actually I don't have any flow that start at that time, so I'm a bit clue less

Here is what I see in the logs prior to the crash:

2 Apr 09:17:11 - [red] Uncaught Exception:
at processImmediate (node:internal/timers:466:21)
at Immediate.<anonymous> (/usr/src/node-red/node_modules/@node-red/runtime/lib/flows/Flow.js:831:52)
2 Apr 09:17:11 - [error] TypeError: sendEvent.destination.node.receive is not a function
'===================
Welcome to Node-RED
2 Apr 09:18:06 - [info]

I also noticed that just after restart there are those errors that I can't neither link to something:

2 Apr 09:18:49 - [warn]
2 Apr 09:18:49 - [info] Starting flows
TypeError: Cannot read properties of undefined (reading 'id')
at Object.start [as startFlows] (/usr/src/node-red/node_modules/@node-red/runtime/lib/flows/index.js:394:33)
at Flow.start (/usr/src/node-red/node_modules/@node-red/runtime/lib/flows/Flow.js:224:41)
at Subflow.start (/usr/src/node-red/node_modules/@node-red/runtime/lib/flows/Subflow.js:209:62)
at Array.map (<anonymous>)
at /usr/src/node-red/node_modules/@node-red/runtime/lib/flows/Subflow.js:209:95
at Array.map (<anonymous>)
at /usr/src/node-red/node_modules/@node-red/runtime/lib/flows/Subflow.js:209:140
2 Apr 09:19:23 - [info] [mongodb-aleph-in:find in custom_metadata] connecting to mongodb...
at Object.start [as startFlows] (/usr/src/node-red/node_modules/@node-red/runtime/lib/flows/index.js:394:33)
at Flow.start (/usr/src/node-red/node_modules/@node-red/runtime/lib/flows/Flow.js:224:41)
at Subflow.start (/usr/src/node-red/node_modules/@node-red/runtime/lib/flows/Subflow.js:209:62)
at Array.map (<anonymous>)
at /usr/src/node-red/node_modules/@node-red/runtime/lib/flows/Subflow.js:209:95
at Array.map (<anonymous>)
at /usr/src/node-red/node_modules/@node-red/runtime/lib/flows/Subflow.js:209:140
TypeError: Cannot read properties of undefined (reading 'id')
2 Apr 09:19:23 - [info] Started flows
(Use `node --trace-warnings ...` to show where the warning was created)

Any advice to help tracking down where this could happen would be appreciated
Thanks

Try setting the log level to Trace in settings.js in the hope it will give more information.

Thanks @Colin

I made the change and will restart the container at night
Then we will see if it crashes again tomorrow

The Trace setting is more about determining what is causing the errors on startup. Though it may give information about the crash too, but less likely.

Ah thanks
However I now have the traces for both

Here are the logs right before the crash

2024-04-04T06:31:37.086Z	4 Apr 06:31:37 - [debug] Flushing localfilesystem context scope 304d6298862b9e4b
2024-04-04T06:34:56.412Z	4 Apr 06:34:56 - [debug] Flushing localfilesystem context scope 304d6298862b9e4b
2024-04-04T06:37:27.891Z	at processImmediate (node:internal/timers:466:21)
2024-04-04T06:37:27.891Z	at Immediate.<anonymous> (/usr/src/node-red/node_modules/@node-red/runtime/lib/flows/Flow.js:831:52)
2024-04-04T06:37:27.891Z	4 Apr 06:37:27 - [error] TypeError: sendEvent.destination.node.receive is not a function
2024-04-04T06:37:27.891Z	4 Apr 06:37:27 - [red] Uncaught Exception:
2024-04-04T06:38:40.550Z	'===================
2024-04-04T06:38:40.550Z	Welcome to Node-RED

I found this Github issue which refers to using node.send()into a promise call back which I'm actually using in a function node:

// Initialize slack
const web = new slackWebApi.WebClient("xxxxxxxxxxxxxxxxxxxxxx");

node.warn(`Slack upload with ${JSON.stringify(msg.payload)}`);

(async () => {
    const result = await web.files.uploadV2(msg.payload);
    msg.payload = result
    node.send(msg)
})();

If this is to be avoided, could you please advise how I should code that instead ?

And the logs during boot issue are those:

2024-04-04T06:39:45.286Z	4 Apr 06:39:45 - [trace] [subflow:be049e0bbf73290d] ------------------|--------------|-----------------
2024-04-04T06:39:45.286Z	4 Apr 06:39:45 - [trace] [subflow:be049e0bbf73290d] start subflow [cdb37cb8bad2f69e/e7ee7117246f4ae3/cab9972dcf655114/09507900c4b343fe]
2024-04-04T06:39:45.286Z	4 Apr 06:39:45 - [trace] [subflow:1cc39ec7a53d287e] start subflow [cdb37cb8bad2f69e/e7ee7117246f4ae3/cab9972dcf655114]
2024-04-04T06:39:45.286Z	4 Apr 06:39:45 - [trace] [subflow:e7ee7117246f4ae3] start subflow [cdb37cb8bad2f69e/e7ee7117246f4ae3]
2024-04-04T06:39:45.288Z	at Object.start [as startFlows] (/usr/src/node-red/node_modules/@node-red/runtime/lib/flows/index.js:394:33)
2024-04-04T06:39:45.288Z	at Flow.start (/usr/src/node-red/node_modules/@node-red/runtime/lib/flows/Flow.js:224:41)
2024-04-04T06:39:45.288Z	at Subflow.start (/usr/src/node-red/node_modules/@node-red/runtime/lib/flows/Subflow.js:209:62)
2024-04-04T06:39:45.288Z	at Array.map (<anonymous>)
2024-04-04T06:39:45.288Z	at /usr/src/node-red/node_modules/@node-red/runtime/lib/flows/Subflow.js:209:95
2024-04-04T06:39:45.288Z	at Array.map (<anonymous>)
2024-04-04T06:39:45.288Z	at /usr/src/node-red/node_modules/@node-red/runtime/lib/flows/Subflow.js:209:140
2024-04-04T06:39:45.288Z	TypeError: Cannot read properties of undefined (reading 'id')
2024-04-04T06:39:45.302Z	4 Apr 06:39:45 - [debug] [subflow:2f9f934b3ee78deb] not starting disabled node : 6388487f96043b52
2024-04-04T06:39:45.302Z	4 Apr 06:39:45 - [trace] [subflow:2f9f934b3ee78deb] start subflow [1ee34aef2976ed38/2f9f934b3ee78deb]
2024-04-04T06:39:45.303Z	4 Apr 06:39:45 - [trace] [subflow:2f9f934b3ee78deb] ------------------|--------------|-----------------

Which seems to locate the problem in subflow e7ee7117246f4ae3
The problem seems to occur in that line (node-red3.0.2):

One of the 2 .id seems to be non existant

Any idea what could cause that ?
We have other node-red instances running in similar conditions (other AWS regions) with same nodes and subflows and I haven't cound the same log in the last 30 days.

Thanks for any hints on those 2 issues

Hello

Anyone with some hints on those 2 errors ?
The most critical one being the daily crash

Thanks

Jean-Michel

The function node is top level async await so this :point_up: is unnecessary.

Something like this would be better (added try-catch to show you options available to you in your function nodes)

// Initialize slack
const web = new slackWebApi.WebClient("xxxxxxxxxxxxxxxxxxxxxx");
node.warn(`Slack upload with ${JSON.stringify(msg.payload)}`);
try {
    const result = await web.files.uploadV2(msg.payload);
    msg.payload = result
    node.send(msg)
} catch (e) {
    node.error(e, msg)
}

FWIW: your original code did not have any error handling inside the IIFE, so it would definitely crash node-red if it failed.

Hi @Steve-Mcl

Many thanks. Still a lot to learn :slight_smile:
Indeed adding error handling code is better

I'll change the code as per your advice and let you know within a couple of days

Best regards

Hi @Steve-Mcl

Wouldn't it be required to add a node.done() ?

try {
    const result = await web.files.uploadV2(msg.payload);
    msg.payload = result
    node.send(msg)
    node.done();
} catch (e) {
    node.error(e, msg)
}

Not really necessary unless you are relying on the complete node.

If node.done() is excluded, it will still (implicitly) trigger a complete node - however it will be triggered regardless of success (i.e. even node.error is called / node.send not called) when the function node ends cleanly. By adding the explicit node.done, you control when the complete node is triggered.

In short, node.send will send regardless of the inclusion or exclusion of node.done

Ok, thanks

Hi @Steve-Mcl @Colin

I did 2 changes and the crash didn't occurred this morning

  1. Remove the async and added the try/catch as suggested by Steve.

  2. There was a flow running daily (from one of my colleague) with a function node doing:

msg.payload = msg.entry.leagues_list;
node.send(msg);
node.done();

msg.payload = msg.entry.teams_list;
node.send(msg);
node.done();

return;

I stopped that flow.
I believe that code is not the best, and may be those 2 x node.done() is bad and more likely responsible for the sendEvent.destination.node.receive is not a function error message.
I believe it should be enough to write it like:

msg.payload = msg.entry.leagues_list;
node.send(msg);
msg.payload = msg.entry.teams_list;
return msg;

Do you aggree ?

Thanks

Why do you even need to send 2 messages? The data you want is already present in the msg. Where are you sending these 2 messages that you generate in the function?

as for "better"? No, what you are doing is better suited to a change nodes and wires (that is after all the point of visual programming)

the wires branching here ^ will cause the msg to be duplicated, then the change nodes will set your payloads - but i re-iterate, since your values are already in one node, why not just use them?

image

Thanks @Steve-Mcl
The visual approach with the 2 changes nodes is definitively the nicest solution

The reason is that there then a split node which create as many messages as elements in both lists.

Another alternative would be to merge both lists into payload and issue a single message to the split node.

"not my flow", I likely would have designed the flow like that anyhow

Thanks again

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.