Ok, that makes sense, since it was the latest page request I had the bad code on.
But if you are using async calls in the NR code, that will spawn threads, will it not? Worker threading? I have read that many times from various sources, it may be in same context, but worker threads must result, in some way, how else would the call back system and Promise objects work otherwise when you use the async modifier?
LOL.. fake threads? Sorry just struck me as a funny thought.
It would help if you could confirm that. Given the infinite loop locks up the runtime it is theoretically impossible for it to any more work (such as handle the second deploy). So at this point I would like to test if the file is being lost as part of the first deploy that introduces the infinite loop.
Ok, I will try to do that. So just to be clear, you want me to confirm flow file exists, trigger the loop, and then do a deploy, and then check for the file again, right?
No. In you description of the steps you take, there are two deploys - step 2 and step 5.
You have only said you check the flow file exists after step 5. I would like you to confirm the file still exists between step 2 and 5.
Ah, got it, ok.
You didn't mention before that you had to trigger the loop, from the description it seemed you were saying that deploying the bad code caused the loop to start automatically.
When you are talking about multiple flows in the editor, do you mean multiple flow tabs you switch between or multiple browser tabs open on different flow tabs?
No, it just uses the event loop intelligently to make it look as though it is multi-threaded. Workers are only just appearing in the latest versions of Node.
It is possible that either Node.js itself or the VM that function nodes run in is clever enough to have a system handoff when an unbound loop is detected. I don't know enough about the inner workings of Node.js
@Colin
Right, just multiple tabs. Some active some not, i.e. one or two had automatic injections running on intervals for example.
The original bad loop was triggered as part of the parsing of HTML returned by an interval injection. So for example, inject, HTML page request/response, parse result, 30 seconds later next inject, page request/response, parse, etc. At the parse step, was the bad loop, so the first initial inject after deploy triggered the processor slamming at the parsing step note above..
One thing in particular I don't understand, I think you said that that you had seen the node (or node-red) process running at 200 or 250% in top. I didn't think that was possible. I never see it more than 100% or thereabouts. You aren't somehow running mulitple instances of node-red are you? Though that shouldn't be possible unless you have them running on multiple ports.
Did you say you are on a PI4 or is it 3? I am trying to get it to fail on a 4.
I captured it, see the top of the thread. The captured data is from a Pi3 but the very first time I suffered the flow file disappearing issue was on a Pi4. If you recall, in that first case, you believed it was how I stopped NR that might have impacted the flow file saving sequence. And... in that very first case, the backup file was still present. If is only in the recent case, this thread documents, that I found that the flow file and the backup of the flow file were missing.
Not running multiple instances, but the top clearly shows node process is over 200% CPU loading. Remember the Pi3 is multi core, so you can see up 400% CPU load or 4.0 loading. The total load was 3.4, the specific node % is 262 or such. So node was slamming the processor, but not consuming everything, call it 2.6 cores maxed out of 4, if you will. As I noted previous, I was still able to open multiple SSH sessions during the loading peak, so the load at 3.4 in effect says that at that time, 1 core still was not fully consumed. When I finally stopped the issue, I actually saw node process go above 310%, I don't recall what the load was, but it had to be pushing up towards 4.0.
This is off topic, but could run multiple instances in docker instances as well, man, that would be interesting on a Pi device!
Side note, since I happen to fire up a 64 bit PC VM running NR, I just fired of the bad loop script test flow, to see what happens. The PC VM is setup with 2 cores, 1 processor.
This is step 1 to building a 32 bit PC NR system, to act as a global monitor of all my Pi devices, etc. But I digress.
Just thought of something... if I can't get the PC version to have the missing files option, this suggests that the write commit handling specific to SSD media might be a factor. The PC VM version is running off a typical old school mechanical disk drive. Moreover, if a write commit fails, the virtualization layer should be throwing all kinds of errors to the VM log.
The PC version, is at times kicking node process CPU % above 100%, I am generating deployments, and I keep getting Error lost connection to server, reconnecting messages, which makes sense. But I am also watching the file system. And so far, the flow files are there. But the time/date stamp on the flow is not changing. So the commits to the file are not happening, again, this makes sense since clearly the server connection is inconsistent under this load. Node is dropping below 100% at times, even though the flow is still running. So the one thing we can say, is the storage subsystem, being different on PC VM versus Pi device, is a factor, in the case of the PC VM, it is handling the load much better over all. PC VM has greater resources, far beyond the Pi devices.
The browser once in a while also reports review of the flow is needed, since there is a difference between content in browser versus file.
Ok, next step is repeat this on a Pi device.
Is there anything in your loop beyond simple processing? Anything that might involve system interaction? Sending messages? Anything else?
I recreated the flow in concept to make it transportable. Here is the current test copy. This should be a fair mimic of the original. I have it running on the PC VM, and later today will test on Pi3 and Pi4, to see if I can get the same behavior I did before.
[{"id":"2e42ee86.478292","type":"tab","label":"Test","disabled":false,"info":""},{"id":"7251a30c.f8eb3c","type":"inject","z":"2e42ee86.478292","name":"Inject","props":[{"p":"payload"}],"repeat":"1","crontab":"","once":true,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":110,"y":60,"wires":[["b7f0c833.0d5738"]]},{"id":"b7f0c833.0d5738","type":"function","z":"2e42ee86.478292","name":"Endless","func":"\nfor (var theIndex = 0; theIndex < 10; theIndex++) {\n \n var theTime = new Date().getTime();\n \n //\n node.warn(theIndex);\n node.warn(theTime);\n}\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":270,"y":60,"wires":[["640f8aec.dd5634"]]},{"id":"640f8aec.dd5634","type":"debug","z":"2e42ee86.478292","name":"Message","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":440,"y":60,"wires":[]}]
That loop doesn't have an infinite loop in it, it just goes round 10 times each time it is triggered (ie every second).
