Hello my friends,
I ve a problem with file synconisation and bigger files. (Node red freezes and restarts)
I tried to sync a 50MB file form Nextcoud / WebDav (@kuehnelbs/node-red-contrib-nextcloud) and my raspi started to lag for some minutes, then NR restarted. Same thing when opening a FTP (node-red-contrib-ftp-server). I checked RAM: 300 MB free on peak. Both nodes give back an array/buffer including the file. Maybe there is a buffer limit or so ... I ve no idea how to fix that or how to transfer bigger files.
Thx for help 
Bump:
I found this as a reason but still have no fix:
<--- JS stacktrace --->
==== JS stack trace =========================================
0: ExitFrame [pc: 0xe8ca40]
Security context: 0x4714e3d1 <JSObject>
1: toJSON [0x45e87e7d] [buffer.js:~1047] [pc=0x21bf6b78](this=0x5f472bd9 <Uint8Array map = 0x29e142c1>)
2: arguments adaptor frame: 1->0
3: InternalFrame [pc: 0xe07db0]
4: EntryFrame [pc: 0xe07ab4]
5: builtin exit frame: stringify(this=0x47145e3d <Object map = 0x29e00a19>,0x51e452a9 <String[#1]: >,0x5f472d5d <JSFunction (sfi = 0x35efca45)>,0x5f472...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
nodered.service: Main process exited, code=killed, status=6/ABRT
nodered.service: Failed with result 'signal'.
nodered.service: Service RestartSec=100ms expired, scheduling restart.
nodered.service: Scheduled restart job, restart counter is at 1.
Stopped Node-RED graphical event wiring tool.
Started Node-RED graphical event wiring tool.
hmmmpf....
well that is out of system memory....
how can that be when transfering a 50 MB file while 600MB ram is free ?
I tried to change /lib/systemd/system/nodered.service
line: Environment="NODE_OPTIONS=--max_old_space_size=256"
to different values. 512 works but doesn´t fix anything.
Does everything work perfectly with smaller files?
Also stop node red and then start it in a terminal so that the startup log is displayed. Post the log from the start to where it fails.
it does work perfect with smaller files.
the log has already been posted (post 2)
Is it trying to handle more than one file at a time ?
When it does handle it it may well duplicate it at some point which would double the requirement. Do you have any forks in your flow ? any debug nodes (that are attached to the main flow) ? etc
(and yes - leave it on 512 if you can)
I asked for the log from startup. Sometimes there can be important information at the start.
After investigating things a bit more and after the hints ( much thx @dceejay !!! ) I found out that removing the debug node will reduce the problem... or better... move the problem to another point:
WebDav (node) puts the downloaded file to a buffer / msg.payload -> Now I need a function (node) to set msg.filename for prepairing -> file node to save it to a wanted path.
The raspi will now run from 600 MB free memory to out of memory if the size is bigger than about 120MB.
Error by file node is: "RangeError: Array buffer allocation failed"
So there is no good reason to eat that much memory in case of such file-size compared to free memory. I think NR will make a view copies of the buffer.... Maybe some are caused by the needed function node ? So I ve pushed the limit a bit but still have no solution for bigger files.... If there is free RAM ... well binding the sucsess of a download to free memory is a bad thing.
The solution would be to put the buffer to chunks / parts and write them to disk. I am just wondering about this kind of problem.... I can not be the first to call for chunked transfers.... hmmmppfff
PS: Expanding the swapfile didn´t works
Show us what you have in the function node please.
this is the Next Cloud /WebDav (@kuehnelbs/node-red-contrib-nextcloud) flow:
[{"id":"36c9b7bc.f44008","type":"tab","label":"Flow 4","disabled":false,"info":""},{"id":"c7b1e5f8.6214f8","type":"nextcloud-webdav-out","z":"36c9b7bc.f44008","wname":"downloadFile","server":"ef9922e1.497078","filename":"","x":810,"y":580,"wires":[["ff51840d.3549f8"]]},{"id":"6f7b1091.8229d8","type":"nextcloud-webdav-list","z":"36c9b7bc.f44008","wname":"getNextcloudFiles","server":"ef9922e1.497078","directory":"/MusikBox/","x":830,"y":340,"wires":[["646b6028.7155a","7573f010.4db7e8"]]},{"id":"a73c022e.27a528","type":"function","z":"36c9b7bc.f44008","name":"dirExistsOnLocal","func":"// compares local Dir and nextcloud dir\n// Will check if a folder with the RfId exist on local...\n// if not so, then forwards the message, else sends no message\nlet ncFiles = msg.payload;\nlet localFiles = msg.localFiles;\n\nfor (let i in ncFiles) \n{\n //console.error(ncFiles[i]);\n //if(typeof(ncFiles[i].basename) === typeof 'undefined') continue;\n if(!ncFiles[i] || !ncFiles[i].basename || !ncFiles[i].basename.match(/\\.zip$/ig)) continue;\n msg.filename = ncFiles[i].filename;\n //msg.something = \"nothing\";\n \n let found = false;\n for (let x in localFiles) \n {\n if (localFiles[x] + \".zip\" === ncFiles[i].basename) \n { \n found = true; \n }\n }\n if(!found) \n { \n msg.download = ncFiles[i].basename;\n msg.parts = \"\";\n return [msg , null];\n } \n\n}\n\nreturn [null, msg];\n\n\n\n//for (i=0; i< len; i++) \n//{\n \n//}\n\n","outputs":2,"noerr":0,"x":830,"y":500,"wires":[["6079bd9d.333c7c","c7b1e5f8.6214f8"],["23f05d7b.e2016a"]]},{"id":"6079bd9d.333c7c","type":"debug","z":"36c9b7bc.f44008","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":1030,"y":440,"wires":[]},{"id":"646b6028.7155a","type":"fs-ops-dir","z":"36c9b7bc.f44008","name":"getLocalFiles","path":"/home/pi/Music/","pathType":"str","filter":"*","filterType":"str","dir":"localFiles","dirType":"msg","x":820,"y":400,"wires":[["a73c022e.27a528"]]},{"id":"f4521eda.42405","type":"debug","z":"36c9b7bc.f44008","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":1030,"y":620,"wires":[]},{"id":"23f05d7b.e2016a","type":"debug","z":"36c9b7bc.f44008","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":1030,"y":580,"wires":[]},{"id":"ced5b8ad.73915","type":"function","z":"36c9b7bc.f44008","name":"isResponseGood","func":"if(msg.responseUrl !== \"https://nc.xxxxxx.de/index.php/login\") return null;\nreturn msg;","outputs":1,"noerr":0,"x":830,"y":280,"wires":[["6f7b1091.8229d8"]]},{"id":"63cd4dba.b8e50c","type":"file","z":"36c9b7bc.f44008","name":"","filename":"","appendNewline":false,"createDir":true,"overwriteFile":"true","encoding":"none","x":830,"y":700,"wires":[[]]},{"id":"ff51840d.3549f8","type":"function","z":"36c9b7bc.f44008","name":"","func":"let targetDir = \"/home/pi/Music/test.zip\";\nmsg.filename = targetDir;\nreturn msg;","outputs":1,"noerr":0,"x":820,"y":640,"wires":[["63cd4dba.b8e50c"]]},{"id":"bddae6f9.780818","type":"inject","z":"36c9b7bc.f44008","name":"","topic":"","payload":"https://nc.xxxxxx.de","payloadType":"str","repeat":"","crontab":"","once":true,"onceDelay":"15","x":810,"y":200,"wires":[["b6c620df.de1c08","4fcf6cbc.435a5c"]]},{"id":"b6c620df.de1c08","type":"http request","z":"36c9b7bc.f44008","name":"","method":"GET","ret":"txt","paytoqs":false,"url":"https://nc.xxxxxx.de","tls":"","persist":false,"proxy":"","authType":"","x":810,"y":240,"wires":[["ced5b8ad.73915"]]},{"id":"7573f010.4db7e8","type":"debug","z":"36c9b7bc.f44008","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":1070,"y":360,"wires":[]},{"id":"4fcf6cbc.435a5c","type":"gc-trigger","z":"36c9b7bc.f44008","display":true,"name":"","x":990,"y":200,"wires":[[]]},{"id":"ef9922e1.497078","type":"nextcloud-credentials","z":"","cname":"XXXXX","address":"https://nc.xxxxxx.de","insecure":false}]
Also testing with FTP server node (node-red-contrib-ftp-server):
[{"id":"4e0dbbfd.623dfc","type":"ftp-server","z":"1edac7e4.ef6288","name":"xxxxxx","port":"2017","x":1080,"y":560,"wires":[["2b7cbed.aeac8c2"]]},{"id":"1d1db6f7.0a81b9","type":"file","z":"1edac7e4.ef6288","name":"","filename":"","appendNewline":false,"createDir":true,"overwriteFile":"true","encoding":"none","x":1530,"y":560,"wires":[[]]},{"id":"2b7cbed.aeac8c2","type":"function","z":"1edac7e4.ef6288","name":"","func":"let targetDir = \"/home/pi/Music/test123.zip\";\nmsg.filename = targetDir;\nreturn msg;","outputs":1,"noerr":0,"x":1290,"y":560,"wires":[["1d1db6f7.0a81b9"]]}]
Does the webdav-out node fetch just one file or a whole directory of files?
in my case only one file.... It can handle a list but I never tried
Does the webdav node pass though other properties correctly ? If so can you attach the filename before calling it so you don't then need to do that afterwards... saving a copy etc...
Does adding a property to the message mean it has to be cloned?
I agree with @Colin -> Adding a property should not clone anything. - Does this lead to a copy of the object (including buffered file == out of memory) !? - Interessting question.
@dceejay I had this thougt too but dropped it because webdav itself uses the property msg.filename. So I cant declare it bevor.
The other bad thing is that 2GB swap seems to be unused.
What does top show?
EDIT: now sometimes webdav dl node comes up with mem allocation error:
click me
You are running mplayer and a linux desktop as well as Node-RED so I'm not at all surprised that you are running out of memory given that Node-RED has also taken 64% of RAM.
It would be more helpful though to sort the htop output by memory use than cpu time.
I am running Mplayer because this is going to be a musicbox project. Killing it is no real scenario. I can reduce space by exiting some stuff like linux desktop...yes, but node red will eat 74% of the memory as you can see at second screen shot.
It will not help freeing up 50 or 100 MB. When a bigger download is triggered after a fresh restart of node red it will eat the 600 MB free mem (I ve afer reboot). The garbage collector will only free up about 300 MB. (Mem leak I think) So the second bigger download will fail. - Doesnt matter if I free up some mem by exiting linux desktop. The problem remains: No chunked downloads possible... I am no expert at this but writing a datastream in small pakages is no new invention.... the file node maybe could do that with "append" option. ... while FTP and webdav nodes do not support this.
