Http-Request - API fatal error

I am using "http request" nodes often in my flows and I have never had any problems.
Right now I am using an http request node to download two CSV files of ~ 2mb size.
When the execution of the flow reaches this node, the blue box is left marked and Node-RED crash without releasing any information.
Restarting Node-RED and deleting the node to put a new one, sometimes it works 1 time but then it keeps failing.
This problem arose after carrying about 3 hours working in the flow and that same node was behaving correctly until its failure.
Environment specifications:

  • Node-RED version: v0.20.5
  • Node.js version: v8.15.1
  • Linux 3.10.0-693.11.1.el7.x86_64 x64 LE
  • Docker image: nodered/node-red-docker
    Node-RED logs:
24/4/2019 14:33:5824 Apr 12:33:58 - [info] Node.js  version: v8.15.1
24/4/2019 14:33:5824 Apr 12:33:58 - [info] Linux 3.10.0-693.11.1.el7.x86_64 x64 LE
24/4/2019 14:33:5824 Apr 12:33:58 - [info] Loading palette nodes
24/4/2019 14:33:5924 Apr 12:33:59 - [warn] rpi-gpio : Raspberry Pi specific node set inactive
24/4/2019 14:33:5924 Apr 12:33:59 - [warn] rpi-gpio : Cannot find Pi RPi.GPIO python library
24/4/2019 14:33:5924 Apr 12:33:59 - [info] Worldmap version 1.5.39
24/4/2019 14:33:59Installing %ccolor:black;font-weight:bold;%c and enabling features %cCLJS DevTools 0.9.10 %ccolor:black %ccolor:#0000ff :formatters color:#ccc :hints color:#ccc :async
24/4/2019 14:33:59BigML NodeRed-Core package
24/4/2019 14:34:0024 Apr 12:34:00 - [info] Dashboard version 2.14.0 started at /ui
24/4/2019 14:34:0024 Apr 12:34:00 - [info] Settings file  : /data/settings.js
24/4/2019 14:34:0024 Apr 12:34:00 - [info] Context store  : 'default' [module=memory]
24/4/2019 14:34:0024 Apr 12:34:00 - [info] User directory : /data
24/4/2019 14:34:0024 Apr 12:34:00 - [warn] Projects disabled : editorTheme.projects.enabled=false
24/4/2019 14:34:0024 Apr 12:34:00 - [info] Flows file     : /data/flows.json
24/4/2019 14:34:0024 Apr 12:34:00 - [info] Server now running at http://127.0.0.1:1880/
24/4/2019 14:34:0024 Apr 12:34:00 - [warn]
24/4/2019 14:34:00
24/4/2019 14:34:00---------------------------------------------------------------------
24/4/2019 14:34:00Your flow credentials file is encrypted using a system-generated key.
24/4/2019 14:34:00
24/4/2019 14:34:00If the system-generated key is lost for any reason, your credentials
24/4/2019 14:34:00file will not be recoverable, you will have to delete it and re-enter
24/4/2019 14:34:00your credentials.
24/4/2019 14:34:00
24/4/2019 14:34:00You should set your own key using the 'credentialSecret' option in
24/4/2019 14:34:00your settings file. Node-RED will then re-encrypt your credentials
24/4/2019 14:34:00file using your chosen key the next time you deploy a change.
24/4/2019 14:34:00---------------------------------------------------------------------
24/4/2019 14:34:00
24/4/2019 14:34:0024 Apr 12:34:00 - [info] Starting flows
24/4/2019 14:34:0024 Apr 12:34:00 - [info] [worldmap:Route map] started at /map
24/4/2019 14:34:0024 Apr 12:34:00 - [info] [inject:startPredict] crontab = 00 06 * * *
24/4/2019 14:34:0124 Apr 12:34:01 - [info] Started flows
24/4/2019 14:34:0124 Apr 12:34:01 - [info] [inject:hostStatus] repeat = 30000
24/4/2019 14:35:0024 Apr 12:35:00 - [info] Stopping modified nodes
24/4/2019 14:35:0024 Apr 12:35:00 - [info] Stopped modified nodes
24/4/2019 14:35:0024 Apr 12:35:00 - [info] Starting modified nodes
24/4/2019 14:35:0024 Apr 12:35:00 - [info] Started modified nodes
24/4/2019 14:36:00
24/4/2019 14:36:00
24/4/2019 14:36:00#
24/4/2019 14:36:00# Fatal error in , line 0
24/4/2019 14:36:00# API fatal error handler returned after process out of memory
24/4/2019 14:36:00#
24/4/2019 14:36:19Illegal instruction (core dumped)
24/4/2019 14:36:19npm ERR! code ELIFECYCLE
24/4/2019 14:36:19npm ERR! errno 132
24/4/2019 14:36:19npm ERR! node-red-docker@1.0.0 start: `node $NODE_OPTIONS node_modules/node-red/red.js -v $FLOWS "--userDir" "/data"`
24/4/2019 14:36:19npm ERR! Exit status 132
24/4/2019 14:36:19npm ERR!
24/4/2019 14:36:19npm ERR! Failed at the node-red-docker@1.0.0 start script.
24/4/2019 14:36:19npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
24/4/2019 14:36:20
24/4/2019 14:36:20npm ERR! A complete log of this run can be found in:
24/4/2019 14:36:20npm ERR!     /root/.npm/_logs/2019-04-24T12_36_19_853Z-debug.log
24/4/2019 14:36:202019-04-24T12:36:26.396857141Z
24/4/2019 14:36:26> node-red-docker@1.0.0 start /usr/src/node-red
24/4/2019 14:36:26> node $NODE_OPTIONS node_modules/node-red/red.js -v $FLOWS "--userDir" "/data"

/root/.npm/_logs/2019-04-24T12_36_19_853Z-debug.log:

0 info it worked if it ends with ok
1 verbose cli [ '/usr/local/bin/node',
1 verbose cli   '/usr/local/bin/npm',
1 verbose cli   'start',
1 verbose cli   '--',
1 verbose cli   '--userDir',
1 verbose cli   '/data' ]
2 info using npm@6.4.1
3 info using node@v8.15.1
4 verbose run-script [ 'prestart', 'start', 'poststart' ]
5 info lifecycle node-red-docker@1.0.0~prestart: node-red-docker@1.0.0
6 info lifecycle node-red-docker@1.0.0~start: node-red-docker@1.0.0
7 verbose lifecycle node-red-docker@1.0.0~start: unsafe-perm in lifecycle true
8 verbose lifecycle node-red-docker@1.0.0~start: PATH: /usr/local/lib/node_modules/npm/node_modules/npm-lifecycle/node-gyp-bin:/usr/src/node-red/node_modules/.bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
9 verbose lifecycle node-red-docker@1.0.0~start: CWD: /usr/src/node-red
10 silly lifecycle node-red-docker@1.0.0~start: Args: [ '-c',
10 silly lifecycle   'node $NODE_OPTIONS node_modules/node-red/red.js -v $FLOWS "--userDir" "/data"' ]
11 silly lifecycle node-red-docker@1.0.0~start: Returned: code: 132  signal: null
12 info lifecycle node-red-docker@1.0.0~start: Failed to exec start script
13 verbose stack Error: node-red-docker@1.0.0 start: `node $NODE_OPTIONS node_modules/node-red/red.js -v $FLOWS "--userDir" "/data"`
13 verbose stack Exit status 132
13 verbose stack     at EventEmitter.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/npm-lifecycle/index.js:301:16)
13 verbose stack     at emitTwo (events.js:126:13)
13 verbose stack     at EventEmitter.emit (events.js:214:7)
13 verbose stack     at ChildProcess.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/npm-lifecycle/lib/spawn.js:55:14)
13 verbose stack     at emitTwo (events.js:126:13)
13 verbose stack     at ChildProcess.emit (events.js:214:7)
13 verbose stack     at maybeClose (internal/child_process.js:915:16)
13 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:209:5)
14 verbose pkgid node-red-docker@1.0.0
15 verbose cwd /usr/src/node-red
16 verbose Linux 3.10.0-693.11.1.el7.x86_64
17 verbose argv "/usr/local/bin/node" "/usr/local/bin/npm" "start" "--" "--userDir" "/data"
18 verbose node v8.15.1
19 verbose npm  v6.4.1
20 error code ELIFECYCLE
21 error errno 132
22 error node-red-docker@1.0.0 start: `node $NODE_OPTIONS node_modules/node-red/red.js -v $FLOWS "--userDir" "/data"`
22 error Exit status 132
23 error Failed at the node-red-docker@1.0.0 start script.
23 error This is probably not a problem with npm. There is likely additional logging output above.
24 verbose exit [ 132, true ]

I have been searching the network, I have verified the cache, I have also used: increase-memory-limit from https://www.npmjs.com/package/increase-memory-limit but I have not achieved anything.

Stopping the service and releasing memory / cache on the host, works once but then, fails again. ÂżIdeas?

As the error says you are running out of memory. It may be that the garbage collection is not happening early enough or you may have something (some node) that is leaking memory.

You can try restricting the memory used before the GC kicks in as per the "passing arguments" section of this doc https://nodered.org/docs/getting-started/running

1 Like

Looks like this ongoing Node.js issue will be improved if not eliminated in Node.js v12 as that version is slated to auto-size on startup.

2 Likes