Node-Red Crashes - Leaves a TCP timeout error prior - How to trace?

29 Mar 19:46:14 - [error] Error: connect ETIMEDOUT 51.195.235.166:443
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1144:16)
nodered.service: Main process exited, code=exited, status=1/FAILURE
nodered.service: Failed with result 'exit-code'. ```


I'm trying to trace this in my project but can't seem to find out what the culprit is.
Any tricks to trace this to which node etc is causing node-red to exit and restart?

Happens about once every minute or two.  Restarts.
NR is responsive until restart.

Is that all the message you get?

What versions of node-red and nodejs (use node -v to get it) are you using?

What hardware/os are you using?

Do you recognise that IP address? The host appears to be vps-e031dc24.vps.ovh.net

Thanks for verifying the host name.. I got as far as ovh.net but nothing specific. Maybe I can trace code to see if there is any entry with that.

My current NR version is 2.1.4
Running on a Linux Ubuntu AWS EC2 service. t3a.medium as of now. I upgraded a step up to make sure I was clear on RAM at least, but never really maxed out even on the config prior...

And the nodejs version?

Sorry: Node Version is v12.22.4

Probably worth upgrading node red to the latest, unless that is a problem for you. Also make sure you are using the latest versions of all installed nodes (check in Manage Palette).

There are two issues

  1. Why is it trying to access that IP, which is presumably something in your flows
  2. Why does the timeout cause a crash. Upgrading node red and all the nodes might help with this one.

You didn't say whether that is the complete message. Can you post the node red log all the way from the startup the beginning of the next restart please.

29 Mar 20:02:21 - [info] Node.js  version: v12.22.4
29 Mar 20:02:21 - [info] Linux 5.4.0-1069-aws x64 LE
29 Mar 20:02:21 - [info] Loading palette nodes
29 Mar 20:02:23 - [info] +-----------------------------------------------------
29 Mar 20:02:23 - [info] | uibuilder initialised:
29 Mar 20:02:23 - [info] |   Using Node-RED's webserver
29 Mar 20:02:23 - [info] |   root folder: /home/ubuntu/.node-red/uibuilder
29 Mar 20:02:23 - [info] |   version . .: 3.3.1
29 Mar 20:02:23 - [info] |   packages . : jquery,socket.io,vue,bootstrap-vue,bootstrap
29 Mar 20:02:23 - [info] +-----------------------------------------------------
29 Mar 20:02:23 - [info] Dashboard version 3.1.2 started at /ui
29 Mar 20:02:24 - [info] Settings file  : /home/ubuntu/.node-red/settings.js
29 Mar 20:02:24 - [info] HTTP Static    : /home/ubuntu/www
29 Mar 20:02:24 - [info] Context store  : 'default' [module=memory]
29 Mar 20:02:24 - [info] User directory : /home/ubuntu/.node-red
29 Mar 20:02:24 - [warn] Projects disabled : editorTheme.projects.enabled=false
29 Mar 20:02:24 - [info] Flows file     : /home/ubuntu/.node-red/flows.json
Unhandled rejection Error: form-data: ENOENT: no such file or directory, open '/02.png'
29 Mar 20:02:24 - [info] Server now running at https://127.0.0.1:443/admin/
29 Mar 20:02:24 - [warn]
---------------------------------------------------------------------
Your flow credentials file is encrypted using a system-generated key.
If the system-generated key is lost for any reason, your credentials
file will not be recoverable, you will have to delete it and re-enter
your credentials.
You should set your own key using the 'credentialSecret' option in
your settings file. Node-RED will then re-encrypt your credentials
file using your chosen key the next time you deploy a change.
---------------------------------------------------------------------
29 Mar 20:02:24 - [info] Starting flows
MongoDB URL: mongodb://127.0.0.1:27017/psiaf2020
MongoDB URL: mongodb+srv://streamtacular:[streamtacular123]@streamtacular.dg46e.mongodb.net/psiaf2020
MongoDB URL: mongodb://127.0.0.1:27017/psiaf2020
connecting:  mongodb://127.0.0.1:27017/psiaf2020
connecting:  mongodb+srv://streamtacular:[streamtacular123]@streamtacular.dg46e.mongodb.net/psiaf2020

... insert a ton of DB connections like logged above...


29 Mar 20:04:37 - [red] Uncaught Exception:
29 Mar 20:04:37 - [error] Error: connect ETIMEDOUT 51.195.235.166:443
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1144:16)
nodered.service: Main process exited, code=exited, status=1/FAILURE
nodered.service: Failed with result 'exit-code'.
nodered.service: Service hold-off time over, scheduling restart.
nodered.service: Scheduled restart job, restart counter is at 27.
Stopped Node-RED graphical event wiring tool.```


I am currently upgrading NR and Node.  I just got done, and I think I yet to have it restart... Will be watching,  and update you later.

Thanks

If you upgraded nodejs make sure it is to v14 or 16, not an odd numbered one. It was the installed nodes that I was suggesting you should check for updates rather than nodejs. nodejs 12 is approaching end of life though, so probably a good thing to update anyway.

Yeah I went ahead and upgraded anyways...
The stable v16...

And NR to the latest...
I am still catching the error...
I will try and update the nodes via the pallet until it boots me...
I may need to look into how to update the nodes via command line.

What is the problem upgrading the nodes via Manage Palette?

Odd:
node -v v16.14.2

but NR reports its using v12

How did you install node-red and nodejs initially? Did you use the recommended Pi/Debian/Ubuntu script Running on Raspberry Pi : Node-RED

How have you upgraded nodejs?

What does
apt policy nodejs
show?

I generally install NR via the instructions at https://nodered.org/docs/getting-started/local
If I remember correctly this is how this instance was setup.

I used nvm to upgrade nodejs

  Installed: 12.22.4-1nodesource1
  Candidate: 12.22.11-1nodesource1
  Version table:
     12.22.11-1nodesource1 500
        500 https://deb.nodesource.com/node_12.x bionic/main amd64 Packages
 *** 12.22.4-1nodesource1 100
        100 /var/lib/dpkg/status
     8.10.0~dfsg-2ubuntu0.4 500
        500 http://us-east-2.ec2.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages
     8.10.0~dfsg-2ubuntu0.2 500
        500 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages
     8.10.0~dfsg-2 500
        500 http://us-east-2.ec2.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages```*emphasised text*

I can confirm all the nodes are updated and just now had a restart... same
It seems to not happen as often?

29 Mar 20:59:45 - [error] Error: connect ETIMEDOUT 51.195.235.166:443
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1144:16)
nodered.service: Main process exited, code=exited, status=1/FAILURE
nodered.service: Failed with result 'exit-code'.
nodered.service: Service hold-off time over, scheduling restart.
nodered.service: Scheduled restart job, restart counter is at 47.
Stopped Node-RED graphical event wiring tool.
Started Node-RED graphical event wiring tool.```

Not a good plan, I believe that installs nodejs in a different place to standard, so now you have both installed. In your position I would remove the nvm version and remove nvm, then I would use the script I linked to, with --node16 on the end (after a space). That should install nodejs 16 and ensure you are on the latest version of node-red.

As to the error, I think you need to find out what in the flows is trying to access that IP address.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.