Node-Red crashes with RangeError: Maximum call stack size exceeded

General
1

Hi, I have a Node-Red instance that crashes from time to time with this error:

kaalut-node-red  | RangeError: Maximum call stack size exceeded
kaalut-node-red  |     at trim_prefix (/usr/src/node-red/node_modules/express/lib/router/index.js:288:23)
kaalut-node-red  |     at /usr/src/node-red/node_modules/express/lib/router/index.js:284:7
kaalut-node-red  |     at Function.process_params (/usr/src/node-red/node_modules/express/lib/router/index.js:335:12)
kaalut-node-red  |     at next (/usr/src/node-red/node_modules/express/lib/router/index.js:275:10)
kaalut-node-red  |     at Layer.handle_error (/usr/src/node-red/node_modules/express/lib/router/layer.js:67:12)
kaalut-node-red  |     at trim_prefix (/usr/src/node-red/node_modules/express/lib/router/index.js:315:13)
kaalut-node-red  |     at /usr/src/node-red/node_modules/express/lib/router/index.js:284:7
kaalut-node-red  |     at Function.process_params (/usr/src/node-red/node_modules/express/lib/router/index.js:335:12)
kaalut-node-red  |     at next (/usr/src/node-red/node_modules/express/lib/router/index.js:275:10)
kaalut-node-red  |     at Layer.handle_error (/usr/src/node-red/node_modules/express/lib/router/layer.js:67:12)

Can you please help me debug this ? Thanks ! Node-Red is 1.0.3 but I already had this bug with pre-1.0 versions.

2

Well first thing to do is to explain what you re doing.

3

Well I have a dozen of web services: HttpIn -> Function -> HttpResponse. Nothing fancy..

4

So what platform are you running NR on?

Was this working previously and if so, what have you added?

5

It's a stock Debian/buster, Node-Red 1.0.3.
It works fine, but after some days/months errors like this appear and I have to restart the docker container.
I'd suspect some kind of leak, but I'm no expert.

6

Ah I've never used Docker but is there a memory option in Docker?

7

By default it uses all available memory. Heres docker stats:

CONTAINER ID        NAME                   CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
1adc26fb5337        kaalut-node-red        2.38%               174.1MiB / 7.615GiB   2.23%               52.8MB / 56.5MB     39.2MB / 45.5MB     23
8

This isn't a lack of memory or other related issue - its a bug where the request has got stuck in a loop and not been able to break out of it. So the fact this is docker etc is irrelevant.

The stack is deep in the route handling of the HTTP In nodes. You say you have lots of HTTP In nodes - can you share more details? Are their paths fixed strings or do they have regex or path parameters in them (eg /foo/:id). Do any of their paths overlap or duplicate each other?

9

All of them have fixed paths e.g. /api/auth/reset.
Only 2 are duplicated in POST hand GET handling nodes (so they share the same path).
But, those flows are unused at the moment.

10

Some flows have Split/Join nodes between the HttpIn and the HttpOut nodes, could this have some consequence ?

11

It happens so frequently we had to set up a cron job to restart the instance, pure despair..
Use case is pretty simple, so it's a show stopper to us :frowning:

12

At least there is a stack trace :slight_smile: Where does it lead ? Thanks !

13

I think everyone is thinking it, so I'll be the first.

1 - How big is the flow?
2 - Do you know how to export things?
3 -

Explain what you mean unused.
4 - Where you are going in/out of the nodes (I have NO understanding of the nodes you refer. (HttpIn -> Function -> HttpResponse.) isolate the links in/out of them and DEPLOY.

See if that stops the problem.

If it does, reconnect them ONE AT A TIME and see where it is failing.

If I was smarter I would show you a flow how to count how many messages are going in/out of the nodes and you could watch which one/s are going crazy.

I may post a small bit soon.

14

Here is a bit of flow you could try:

NODES REQUIRED!
node-red-contrib-counter

[{"id":"f58a9968.916c38","type":"inject","z":"1d85b106.5dfcd7","name":"EXISTING NODE","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":190,"y":2540,"wires":[["daa71ef4.9a7698"]]},{"id":"daa71ef4.9a7698","type":"counter","z":"1d85b106.5dfcd7","name":"","init":"0","step":"1","lower":"","upper":"","mode":"increment","outputs":2,"x":390,"y":2540,"wires":[["67b34c28.e72bdc"],["afb22c35.43f168"]]},{"id":"afb22c35.43f168","type":"debug","z":"1d85b106.5dfcd7","name":"EXISTING NODE","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":600,"y":2540,"wires":[]},{"id":"67b34c28.e72bdc","type":"debug","z":"1d85b106.5dfcd7","name":"WATCH THIS SPACE!","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":610,"y":2480,"wires":[]}]

Screenshot from 2020-03-01 18-24-10
Screenshot from 2020-03-01 18-24-10742×207 11.7 KB

So, the idea:
The two nodes EXISTING NODE are what exist. They are there for clarity.
You simply put the counter and WATCH THIS SPACE! node between the existing node and the http node thingy.

I would also suggest you append a number to the name of the watch this space! node so you know who is counting.

Then, if you get a run away bit of code, you can see where it is happening.

It won't fix the problem, but it may help find where it is happening.

15

What is happening between the split and join ? Could the join be leaving fragments of incomplete messages behind ?

16

This isn't a memory leak.

The stack shows it is stuck in a loop in the express route handler.

I can't immediately think of how to get more information on that without going in and adding some console.log statements around the lines of code from the stack to identify what route exactly it's stuck handling.

17

I can add console log messages, would you have an example ? Thanks !

18

I would start by adding:

 console.log(path);

to line 289 of /usr/src/node-red/node_modules/express/lib/router/index.js so it should look like:

288   function trim_prefix(layer, layerError, layerPath, path) {
289     console.log(path); // <<< ADD THIS LINE
290     if (layerPath.length !== 0) {
291       // Validate path breaks on a path separator
292       var c = path[layerPath.length]
293       if (c && c !== '/' && c !== '.') return next(layerError)

That will generate a lot of noise in the log output, but it should be very clear what path it was trying to process when it got stuck in a loop.

19

Ok, so it displays a path, but what next ? Thanks

20

The reason for doing this was to narrow down what path is causing the issue. You now know the path, so the next steps depends on what it has told you.

Does that path look like something you recognise from your application?
Does it narrow down what route is causing the issue?

Can you share what the path is as well as how you have any HTTP In nodes configured to handle that path?