I have a mission ciritical NodeRed application running on version 2.0.6 that has been running for a couple of years with no obvious issues. The same application is running at a number of sites and the incoming communication is via TCP sockets. There are a number of API calls to external applications using HTTP request Nodes.
Recently one of the sites has started freezing randomly and sometimes automatically restarting output, it will also usually restart if I hit Enter on the powershell console. The application does not request any input, and I am not seeing any errors being logged to either the console or the debug window. It simply freezes. There does not seem to be any pattern as to where it stops.
The host PC is a Windows 10 Pro 20H2 build, the same as other sites.
I have not made any updates to either the NodeJS environment or NodeRed, so I dont believe any of the libraries have changed.
I have added a 5s heartbeat poll to one of the sockets to constantly monitor the status via node.warn, and that also freezes when the application stops.
Any ideas would be welcome..
If it is on a Windows OS, you should check the Windows Event Logs as well to see if there is some external event happening at the time it freezes.
Other than that, you might need to up the logging output maybe right up to trace level. Though you may need to manage the log sizes.
You could also run Windows Resource Monitor to see if anything crazy is happening to resources (CPU, memory, swap, GPU, network).
Also, how are you executing Node-RED on that device?
Thanks for your quick response;
I have been monitoring the Windows Application and System event logs, and I cannot see anything suspicious that correlates with the freezing.
When the problem was initially reported I found that the microsoft Store app was using high CPU, so I have permanently disabled that, but I do not think it was related, CPU load is now normally under 15%.
I should have mentioned that I also modified settings.js to log all warn level messages to a text file, this has been in place for about a year, so I do not believe it is related. Note that the text logging also freezes at the same time as the console.
This site is currently running Node red from the command line in a powershell window.
I also have sites that are running via Electron.
I think I have found the root cause of the issue, this VM has started triggering Kernel Event 16 about 2 weeks ago, clearing the access History in hive. I am yet to figure out why this started occurring, but it looks as if node-red unfreezes each time this event triggers.
Unfortunately the Kernel events are not the issue, we have found that they have been occurring since the VM was created over a year ago.
The only consistent behaviour seems to be the the application will restart if I hit CR from the powershell console. It is as if the program is expecting input, but there are no nodes programmed for user input.
When it freezes can you still get into the node red editor in the browser?
Is this Windows 10, 11 or server?
Are you using the "new" Windows terminal or a straight PS shell? And which version of PS?
Also, did you install any nodes as admin?
As long as the VM still has resource headroom, I would recommend at least a workaround of running Node-RED under PM2 to see if that helps. Ultimately though, you probably want to use something like nssm to turn Node-RED into a server task.
Thanks again for your response;
As per my original post, the OS is Windows10 20H2
I would have installed all nodes as the engineer user, who has admin rights.
I am using a standard Windows Powershell.
Normally once I have the application stable for a site, I move the node-red project to an Electron build for that client, and run as an electron exe after that.
This site has not yet been migrated to Electron, and is running simply in powershell.
I raised the issue, because I have never seen this freeze situation in the past.
It is more whether you installed using an admin PowerShell as opposed to a normal one. Did you install Node-RED with a -g switch (which you can though if aiming to convert to Electron you probably should not) and did you install any nodes that way (which you should not)?
Then how do you run Node-RED? Which user or SYSTEM account starts it and how?
Have you tried running under PM2?
Here is a bit of pain I've experienced in the past... maybe it relates to what you are seeing.
It sounds like you are starting node-red from an interactive powershell, yes? I have seen some applications running inside a terminal window "freeze up" whenever I highlight a portion of the text shown in the interactive window. If hitting the esc key a few times causes node-red to continue running, then you may have the same situation happening...
BTW, the "selected text" may not even show up visually, if the mouse is clicked on whitespace or selected a gap between words (if that makes sense). Whether this is your issue or not, i would suggest you use some other "background service" setup to run node-red instead of using an interactive shell. Or better yet, use Linux!
