Node red crash. - JavaScript heap out of memory

General
1

Hello Volks.

I have developed a Node red application and version control it on github. I have used the project feature and my version controlled files are like below:
[flows_raspberrymongo.json]
[flows_raspberrymongo_cred.json]
package.json

In my application I have used several nodes using configuration, for example mongodb nodes etc.My application work fin on the raspberry I have developed it on. I have set up a Digital Ocean environment for the later purpose of the application, and a remote mongo DB on another digital ocean instance (Firewall openingas and mongo users are check OK)

My problem is. When I clone my project on my "new digital ocean" instance, It hangs for minutes and spit out stack trace according to below. out of Memory ... Then I do som ekill -9 on the node red process, and start in "node-red --safe" and then I can modify my config setting on remote DB so it point correct. If I then deploy, my application start. All working fine (in the safe mode)access to DB and everything response quick.

BUT - as soon I try to start NR in normal mode, then It hangs again and crash...

I guess it has something to do with some configuration. Do I need to version control the _cred.json file? is the .config.json dynamically built up? what i the minimum I need to version control in git to get it up (if _cred fil is build up dinamically if I configure my mngodb2 node I have no problem with that...)

Thanks for help.

5 Mar 14:52:16 - [info] Starting flows
<--- Last few GCs --->
[26144:0x55aeb9d3c400] 73069 ms: Mark-sweep 263.6 (327.8) -> 263.5 (296.8) MB, 321.6 / 0.0 ms (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 449 ms) last resort GC in old space requested
[26144:0x55aeb9d3c400] 73530 ms: Mark-sweep 263.5 (296.8) -> 263.5 (296.8) MB, 461.5 / 0.0 ms last resort GC in old space requested
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0xc171e798fe1
2: /* anonymous */ [/usr/local/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/flows/index.js:350] [bytecode=0x8665d892749 offset=9](this=0x379fbf1880a9 ,nid=0x22fb187e9831 <String[14]: 2b29c038.1cfdb>)

3: arguments adaptor frame: 3->1
4: forEach(this=0x16e555ad791 <JSArray[159]>)
5: start [/usr/local/lib/node_modules/node-red/node_modules/@node...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [node-red]
2: 0x55aeb92f3011 [node-red]
3: v8::Utils::ReportOOMFailure(char const*, bool) [node-red]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node-red]
5: v8::internal::factory::NewFixedArray(int, v8::internal::PretenureFlag) [node-red]
6: v8::internal::HashTable<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::New(v8::internal::Isolate*, int, v8::internal::PretenureFlag, v8::internal::MinimumCapacity) [node-red]
7: v8::internal::HashTable<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::EnsureCapacity(v8::internal::Handlev8::internal::NameDictionary, int, v8::internal::PretenureFlag) [node-red]

And - some restart it thows out DB exception...
2.xx.xxx:27017 timed out]
5 Mar 15:59:08 - [error] [mongodb2 in:insert] MongoError: failed to connect to server [165.22.xx.xxx:27017] on first connect [MongoError: connection 0 to 165.22.xx.xxx:27017 timed out]
5 Mar 15:59:08 - [error] [mongodb2 in:dynamic operation] MongoError: failed to connect to server [165.22.xx.xxx:27017] on first connect [MongoError: connection 0 to 165.22.xx.xxx:27017 timed out]
5 Mar 15:59:08 - [error] [mongodb2 in:dynamic operation] MongoError: failed to connect to server [165.22.xx.xxx:27017] on first connect [MongoError: connection 0 to 165.22.xx.xxx:27017 timed out]
... hundreds of them...
But, after doing trivial chnage like moving a log nod e2 mm and redeploy, everything works "again"

1 Like
2

So you managed to make node.js run out of heap space and it couldn't recover any back using the garbage collection process.

You haven't given us much to go on but the telling comment for me is that it seems to hang for several minutes. That kind of behaviour is nearly always a TCP/IP timeout issue. Or at least that is the first thing to look at.

So check the settings. Make sure that your DO environment allows you to actually reach the MongoDB instance.

3

Hello

Thanks for answer.

It is strange. When I start NR in --safe mode, and manually deploy, I got connection to DB and it is running. But, when start in "node-red-start" mode It hangs, but after some retry I could get it up. I can see I have connection to DB.

020-03-05T19:13:32.403+0000 I NETWORK [conn102] end connection 134.209.2xx.xx:48892 (5 connections now open)
2020-03-05T19:13:32.403+0000 I NETWORK [conn106] end connection 134.209.2xx.xx:48902 (4 connections now open)
2020-03-05T19:13:32.403+0000 I NETWORK [conn103] end connection 134.209.2xx.xx:48894 (3 connections now open)
2020-03-05T19:14:59.195+0000 I NETWORK [listener] connection accepted from 134.209.2xx.xx:48928 #107 (4 connections now open)
2020-03-05T19:14:59.199+0000 I NETWORK [conn107] received client metadata from 134.209.2xx.xx:48928 conn: { driver: { name: "nodejs", version: "2.2.36" }, os: { type: "Linux", name: "linux", architecture: "x64", version: "4.15.0-88-generic" }, platform: "Node.js v8.10.0, LE, mongodb-core: 2.1.20" }
2020-03-05T19:14:59.251+0000 I ACCESS [conn107] Successfully authenticated as principal nodered on TEST1
2020-03-05T19:15:08.944+0000 I NETWORK [conn107] end connection 134.209.2xx.xx:48928 (3 connections now open)
2020-03-05T19:17:05.271+0000 I NETWORK [listener] connection accepted from 134.209.2xx.xx:48932 #108 (4 connections now open)
2020-03-05

But, suddenly NR crash with several

5 Mar 19:18:38 - [error] [mongodb2 in:dynamic operation] MongoError: failed to connect to server [165.22.xx.xxx:27017] on first connect [MongoError: connection 0 to 165.22.xx.xxx:27017 timed out]

I switched to a DB on 127.0.0.1 and experienced the same behaviour. So, I do not think it is TCP/IP related.

To help me understand. What happens if I delete the _cred file, and manually reenter the config to the DB - Is that a good idea? (to be sure there is no settings somewhee from the git checked in environment that was not overwritten by my change on the mongo3 driver node?

Should I try to remove the .config.json file as well to try?

Thansk for you help?

Is there more log file I could send you - which ones are you onterested to see?

Thanks Erik

4

You can rename your cred and the flow files then restart Node-RED, it will recreate blank ones for you. That would be what I'd do next.

5

Hello.

I run NR in --safe mode. I configure mongodb2 config to point to a local mongoDB (which is installed locally as well )
. Switched config to to point to 127.0.0.1:27017
If I use an unvalid username pwd - I got this:

6 Mar 07:03:29 - [info] Started modified nodes
6 Mar 07:03:29 - [error] [mongodb2 in:Insert] MongoError: Authentication failed.
6 Mar 07:03:29 - [error] [mongodb2 in:Insert] MongoError: Authentication failed.

Thats great. expected

I I swop to an authorized user, My application is working GREAT... for the moment.

log from mongodb22020-03-06T07:03:29.037+0000 I NETWORK [conn188] end connection 127.0.0.1:34058 (0 connections now open)
2020-03-06T07:04:00.238+0000 I NETWORK [listener] connection accepted from 127.0.0.1:34060 #189 (1 connection now open)
2020-03-06T07:04:00.240+0000 I NETWORK [conn189] received client metadata from 127.0.0.1:34060 conn: { driver: { name: "nodejs", version: "2.2.36" }, os: { type: "Linux", name: "linux", architecture: "x64", version: "4.15.0-88-generic" }, platform: "Node.js v8.10.0, LE, mongodb-core: 2.1.20" }
2020-03-06T07:04:03.962+0000 I STORAGE [conn189] createCollection: TEST1.paymentoptions with generated UUID: 83cf2284-e105-4246-aaeb-d16aad802f18

I stop NR, and start it in "normal mode -v , but verbose flag on"

And, loads (I assume one exception per mongodb2 node I have in may flows, e.g. around 100) of exceptions of this kind:

6 Mar 07:12:23 - [info] [inject:trigger every day] crontab = 00 12 * * *
6 Mar 07:13:22 - [info] [inject:51a2ec64.d4bee4] repeat = 60000
6 Mar 07:13:30 - [info] Started flows
6 Mar 07:13:30 - [error] [mongodb2 in:Insert] MongoError: failed to connect to server [127.0.0.1:27017] on first connect [MongoError: connection 0 to 127.0.0.1:27017 timed out]
at Pool. (/home/erik/.node-red/node_modules/mongodb-core/lib/topologies/server.js:336:35)
at emitOne (events.js:116:13)
at Pool.emit (events.js:211:7)
at Connection. (/home/erik/.node-red/node_modules/mongodb-core/lib/connection/pool.js:280:12)
at Object.onceWrapper (events.js:317:30)
at emitTwo (events.js:126:13)
at Connection.emit (events.js:214:7)
at Socket. (/home/erik/.node-red/node_modules/mongodb-core/lib/connection/connection.js:199:10)
at Object.onceWrapper (events.js:313:30)
at emitNone (events.js:106:13)
at Socket.emit (events.js:208:7)
at Socket._onTimeout (net.js:420:8)
at ontimeout (timers.js:482:11)
at tryOnTimeout (timers.js:317:5)
at Timer.listOnTimeout (timers.js:277:5)

NR is, althoug all exception starting up. But NO db connection of course. IF I move a log nod x position (trivial change) and deploy, 2 seconds later IT IS WORKING, no problem at all with DB connectiion.

SO - I assume my problem is that there is another set of something that is used at NR start up. But, if I go i --safe mode and deploy manually, these settings are overridden by something that is "better" and obviously working.

Have you guys experienced similar before?

Thanks

I also. stopped, renamed the_cred.json file. , restart. No effect. It was created a new _cred file and after some config it was populated with some little data. That had no effect. Sam behaviour.

Many thanks for support.

/Erik

6

Hello again.

My application has about 30 flows. If all of them activie, the above error will always be reproduced when starting in normal mode (using -verbos flag)

IF i disable som 15 flows, I CAN start the application without db ERROR. Then I slowly enable flow 16, 17 one by one. And suddenly it start to crash (according to above, failed to connect)

I thought maybe it was "specifically on flow "16" it was a problem, but after some trying it appears that is seems to be a tipping poing when the time from "Starting Flows" to "started flows" exceed 30 seconds that the problem appears?

On this one:

6 Mar 09:31:15 - [info] Starting flows
6 Mar 09:31:47 - [info] Started flows

that was 32 seconds. It CRASHED.

I disabled some random flows, deployed (yes, that is no problem) cmd-c to stop, node-red -v
6 Mar 09:38:20 - [info] Starting flows
6 Mar 09:38:50 - [info] Started flows
6 Mar 09:38:50 - [info] [mqtt-broker:Mosquitto on 134] Connected to broker: mqtt://127.0.0.1:1883

NO PROBLEM.

So - is there a secuential thing when connection to DB? So NR start to conenct, and need all flows to be up until conection is finalized. IF the loadtime to load the flows take more than 30 seconds - There is some time out???? Could this possibly a scenario????

Error in the GUI is:
"MongoError: failed to connect to server [127.0.0.1:27017] on first connect [MongoError: connection 0 to 127.0.0.1:27017 timed out]"

------ A little later ----
I found this post:

I have an application in Node.JS which basically retrieves a snapshot of data from a WebAPI each 30 minutes. When I try to interrogate the database by using an aggregate query, the app triggers an error related to timeout (MongoError: connection 1 to 127.0.0.1:27017 timed out). From my logs I see that it's exactly 30 seconds. The aggregate query is something like this:

Maybe it is similar thing..
Is there a way to adjust the time out so it exceed the start up time of the flows?

Thanks Erik

7

Looks like you are getting to the problem.

I'm afraid that I no longer use MongoDB for anything as I had too many issues with it. Most of what I want to do is time-based anyway so InfluxDB was a better fit for me. So I'm rather limited in how I can help.

Not sure about timeouts. Can you adjust the configuration of MongoDB itself?

Also, are you making different MongoDB connections on those different tabs? If so, maybe try to have a single connection node with a link-in node in front and then use the same output node for all of your flows. See if that helps.

8
  1. which mongodb nodes do you use? There’s several in the flows library.
  2. Are you making sure that your mongo nodes use the same configuration node with the connection? Otherwise you keep opening new connections to your database.
  3. How is the database configured in terms of incoming connections, maximum number of connections in the pool, timeout values and so on.
9

Hello

Thanks for help...

So, you were basically right Totallyinformation... TCP IP problem.

MongoDB has a default connections time out of 30 secs. By adding parameter after the Mongodb URI
mongodb://165.22.xx.xxx:27017/TEST1?socketTimeoutMS=120000&connectTimeoutMS=120000

I tell mongo keep the connection open for 120 sec. My application take 45 sec to load.

(I could not really understand how these javascript heap memory exceptions come across... anyway)

So . For the community:

IF you have a large application take longer than 30 sec to load. Adjust the socketTimeoutMS and connectTimeoutMS to MongoDB.

Have a nice day!

/Erik

4 Likes
10

Phew, just shows that the last 40 years in IT hasn't been all for nothing! :grinning:

3 Likes
11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.