Recover from server crash

You have highlighted a particularly hard to solve issue - where an error is thrown in an asynchronous callback the user has created without any proper error handling. We cannot add error handling for that because it isn't our code.

In your code you know when the server crashes

4 Sep 08:22:55 - [red] Uncaught Exception:
4 Sep 08:22:55 - ReferenceError: undefindenFunction is not defined
at Function node:cdf5d431.98a838 [Camera Init]:14:5
nodered.service: Main process exited, code=exited, status=1/FAILURE

however a new user mite be confused by the message "This site can’t be reached"
instead of leaving the user in mid air with a dead server you can redirect to a page that tels the user what happened and present him with some options

image

I think it is quite nice that you actually see the problem presented to you why it crashed.

It crashed - it cannot give you any options, resolve your code - it is part of programming.

THe problem here, as I told you in the github issue, is that we don't know what has caused the crash.

It could be:

  1. a problem in the user's own code - such as the example you gave
  2. a bug in any of the nodes they have installed
  3. a bug in the core of node-red

There is literally no way for us to know what has caused it.

Again, as I replied in the issue, node.js is very clear on what to do with uncaughtExceptions:

From node's own documentation on handling 'uncaughtExceptions'

Unhandled exceptions inherently mean that an application is in an undefined state. Attempting to resume application code without properly recovering from the exception can cause additional unforeseen and unpredictable issues.

The correct use of 'uncaughtException' is to perform synchronous cleanup of allocated resources (e.g. file descriptors, handles, etc) before shutting down the process. It is not safe to resume normal operation after 'uncaughtException' .

So as much as I agree it would be nice to provide more feedback, there is no reliable way for us to do so.

At best we could add more to the error message in the log output to point the user at possible causes. But we certainly couldn't keep the server running just to present a webpage. If you are in production, you'd want the system to restart quickly instead of serve a webpage.

1 Like

after crash nodered logs:

4 Sep 08:22:55 - [red] Uncaught Exception:
4 Sep 08:22:55 - ReferenceError: undefindenFunction is not defined
at Function node:cdf5d431.98a838 [Camera Init]:14:5
nodered.service: Main process exited, code=exited, status=1/FAILURE

so you do know what crashed and you can tell this to the user:

 function node:  **[Camera Init]** caused a crash
 reason: ReferenceError: undefindenFunction is not defined
at Function

nodered.service: Unit entered failed state.

failed state could present a web page that displays the above information and the complete log for more information
in the .node-red folder you have .flows_raspberrypi.json.backup
you could present the user a button that restores the backup and present other ways to exit the failed state or even teach him how to use try/catch

user experience could better as long as the user does not see the default page not found of the browser.
nodered is about simplicity and ease of use and leaving a newcomer that opened noederd for the first time disconnected form the server for a typo is not something that you want for new users.

If you are in production, you'd want the system to restart quickly instead of serve a webpage.

good point here, but pushing a kid or his teacher into the terminal is not a good experience either

For the very specific example of a Function node not handling its errors, then we might be able to detect that and provide a more detailed error in the log - as I said in my previous reply.

No. That is the service wrapper to node-red that is used as part of the raspberry pi install - it is not Node-RED itself. 'failed state' means the service is not running - there it nothing running to serve the page.

I agree, but none of your proposals are practical to actually implement.

1 Like

In my humble opinion a node-red user should not see "page not found" or have to use any other kind of tools to restore functionality of the node-red server being the fact that all he/she did was a typo

Imagine that windows would BSOD on you every time you type a letter in the calculator app.

angular comes with development flag by default
node-red could have a toggle switch for development / production

Careful what you imagine. :stuck_out_tongue_closed_eyes:

3 Likes

so far it does not .... will test tomorrow ... maybe it will :slight_smile:

It is not only the 'uncaughtException', but every async operation like unhandled errors in EventEmitters (missing on('error',...)) and unhandled promise rejections. If any of these happen in your function node, in some cases you cannot even tell that it came from there, as the code is executed from the event loop.

As @knolleary stated, it is hard to determine, where the problem has occurred exactly. So the only reasonable thing is to let the process crash and perform an automatic restart by some process manager like pm2, nssm, etc - at least for production systems. Otherwise you could end up with inconsistent states.

2 Likes

Did someone say calculator crashes.... ? only occasionally - https://www.google.com/search?q=crash+windows+calculator+app&oq=crash+windows+calculator+app&aqs=chrome..69i57.6184j0j7&sourceid=chrome&ie=UTF-8

:slight_smile:

3 Likes

no BSOD .... so it does not apply to this topic ... you will have to dig deeper :))))

No bsod but just bails out.

1 Like

Sorry, late to this discussion but if the error happens inside a function node, doesn't that crash the VM? If so, can't the VM itself be wrapped in a try/catch? Not sure that would catch everything but it might help?

Sorry if that is just too simplistic, I've not had a chance to look at the core code where the VM is set up.


Also, am I the only one who spotted the typo in the error message? "seams" should be "seems" - is that message from the browser or from Node-RED?

@TotallyInformation if it were that easy we would have already done that....

I cannot see any error message in this thread that contains either 'seams' or 'seems'. Where are you looking?

NOTE:
Dyslexia is a serious condition for those who suffer, or cannot access spell checkers.

I think I know what the problem is... It's probably the "undefindenFunction" its not defindened... what is this undefinden Function? perhaps its just a reference that's spelled wrongly. Perhaps you should just definden it...

dyslexia sukcs!

image

2 Likes

In the screenshot from the web page

That image is a mock-up example made by @dsl400. It is not a real image of any message we produce - it was a suggestion for the type of thing we could do.

There does not need to be any further critique over the contents of the image. It is not practical for us to display any such error message in the browser.

sorry ! :face_with_raised_eyebrow:

I thought it was funny... :stuck_out_tongue_closed_eyes: (because I have dyslexia... :sob: )

Sorry, I hadn’t recognised it as a mockup, thought it was the actual error shown but reading this topic again carefully obviously makes clear no error page would have been shown. The realistic looking error page got me confused.

2 Likes

it is exactly what you say it is, a typo and if you read the whole discussion you can see that this exactly the subject of this thread, a typo can stop the node-red server

funny! isn't it ?!