# File transfers fail because of file size - buffer limit?

**URL:** <https://discourse.nodered.org/t/file-transfers-fail-because-of-file-size-buffer-limit/20601>\
**Category:** General\
**Created:** [19 January 2020 22:39 UTC](https://discourse.nodered.org/t/file-transfers-fail-because-of-file-size-buffer-limit/20601 "2020-01-19T22:39:30Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![WhiteLion](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/whitelion/32/12266_2.png) [@WhiteLion](https://discourse.nodered.org/u/WhiteLion)\
**Post date:** [19 January 2020 22:39 UTC](https://discourse.nodered.org/t/file-transfers-fail-because-of-file-size-buffer-limit/20601/1 "2020-01-19T22:39:30Z")

</div>

Hello my friends,  
I ve a problem with file synconisation and bigger files. (Node red freezes and restarts)  
I tried to sync a 50MB file form Nextcoud / WebDav (@kuehnelbs/node-red-contrib-nextcloud) and my raspi started to lag for some minutes, then NR restarted. Same thing when opening a FTP (node-red-contrib-ftp-server). I checked RAM: 300 MB free on peak. Both nodes give back an array/buffer including the file. Maybe there is a buffer limit or so ... I ve no idea how to fix that or how to transfer bigger files.  
Thx for help 🙂

---

<div class="post-metadata">

**Author:** ![WhiteLion](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/whitelion/32/12266_2.png) [@WhiteLion](https://discourse.nodered.org/u/WhiteLion)\
**Post date:** [20 January 2020 21:39 UTC](https://discourse.nodered.org/t/file-transfers-fail-because-of-file-size-buffer-limit/20601/2 "2020-01-20T21:39:13Z")

</div>

Bump:  
I found this as a reason but still have no fix:

```auto

<--- JS stacktrace --->
==== JS stack trace =========================================
    0: ExitFrame [pc: 0xe8ca40]
Security context: 0x4714e3d1 <JSObject>
    1: toJSON [0x45e87e7d] [buffer.js:~1047] [pc=0x21bf6b78](this=0x5f472bd9 <Uint8Array map = 0x29e142c1>)
    2: arguments adaptor frame: 1->0
    3: InternalFrame [pc: 0xe07db0]
    4: EntryFrame [pc: 0xe07ab4]
    5: builtin exit frame: stringify(this=0x47145e3d <Object map = 0x29e00a19>,0x51e452a9 <String[#1]: >,0x5f472d5d <JSFunction (sfi = 0x35efca45)>,0x5f472...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
nodered.service: Main process exited, code=killed, status=6/ABRT
nodered.service: Failed with result 'signal'.
nodered.service: Service RestartSec=100ms expired, scheduling restart.
nodered.service: Scheduled restart job, restart counter is at 1.
Stopped Node-RED graphical event wiring tool.
Started Node-RED graphical event wiring tool.

```

hmmmpf....

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [20 January 2020 22:09 UTC](https://discourse.nodered.org/t/file-transfers-fail-because-of-file-size-buffer-limit/20601/3 "2020-01-20T22:09:42Z")

</div>

well that is out of system memory....

---

<div class="post-metadata">

**Author:** ![WhiteLion](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/whitelion/32/12266_2.png) [@WhiteLion](https://discourse.nodered.org/u/WhiteLion)\
**Post date:** [20 January 2020 22:12 UTC](https://discourse.nodered.org/t/file-transfers-fail-because-of-file-size-buffer-limit/20601/4 "2020-01-20T22:12:23Z")

</div>

how can that be when transfering a 50 MB file while 600MB ram is free ?

I tried to change `/lib/systemd/system/nodered.service`  
line: `Environment="NODE_OPTIONS=--max_old_space_size=256"`  
to different values. 512 works but doesn´t fix anything.

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [20 January 2020 22:26 UTC](https://discourse.nodered.org/t/file-transfers-fail-because-of-file-size-buffer-limit/20601/5 "2020-01-20T22:26:41Z")

</div>

Does everything work perfectly with smaller files?  
Also stop node red and then start it in a terminal so that the startup log is displayed. Post the log from the start to where it fails.

---

<div class="post-metadata">

**Author:** ![WhiteLion](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/whitelion/32/12266_2.png) [@WhiteLion](https://discourse.nodered.org/u/WhiteLion)\
**Post date:** [20 January 2020 23:06 UTC](https://discourse.nodered.org/t/file-transfers-fail-because-of-file-size-buffer-limit/20601/6 "2020-01-20T23:06:04Z")

</div>

it does work perfect with smaller files.  
the log has already been posted (post 2)

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [20 January 2020 23:25 UTC](https://discourse.nodered.org/t/file-transfers-fail-because-of-file-size-buffer-limit/20601/7 "2020-01-20T23:25:01Z")

</div>

Is it trying to handle more than one file at a time ?  
When it does handle it it may well duplicate it at some point which would double the requirement. Do you have any forks in your flow ? any debug nodes (that are attached to the main flow) ? etc  
(and yes - leave it on 512 if you can)

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [21 January 2020 07:32 UTC](https://discourse.nodered.org/t/file-transfers-fail-because-of-file-size-buffer-limit/20601/8 "2020-01-21T07:32:40Z")

</div>

> [@WhiteLion](#):
>
> the log has already been posted

I asked for the log from startup. Sometimes there can be important information at the start.

---

<div class="post-metadata">

**Author:** ![WhiteLion](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/whitelion/32/12266_2.png) [@WhiteLion](https://discourse.nodered.org/u/WhiteLion)\
**Post date:** [21 January 2020 09:34 UTC](https://discourse.nodered.org/t/file-transfers-fail-because-of-file-size-buffer-limit/20601/9 "2020-01-21T09:34:06Z")

</div>

After investigating things a bit more and after the hints ( much thx @dceejay !!! ) I found out that removing the debug node will reduce the problem... or better... move the problem to another point:  
WebDav (node) puts the downloaded file to a buffer / msg.payload -\> Now I need a function (node) to set msg.filename for prepairing -\> file node to save it to a wanted path.  
The raspi will now run from 600 MB free memory to out of memory if the size is bigger than about 120MB.  
Error by file node is: "RangeError: Array buffer allocation failed"

So there is no good reason to eat that much memory in case of such file-size compared to free memory. I think NR will make a view copies of the buffer.... Maybe some are caused by the needed function node ? So I ve pushed the limit a bit but still have no solution for bigger files.... If there is free RAM ... well binding the sucsess of a download to free memory is a bad thing.  
The solution would be to put the buffer to chunks / parts and write them to disk. I am just wondering about this kind of problem.... I can not be the first to call for chunked transfers.... hmmmppfff

PS: Expanding the swapfile didn´t works

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [21 January 2020 10:00 UTC](https://discourse.nodered.org/t/file-transfers-fail-because-of-file-size-buffer-limit/20601/10 "2020-01-21T10:00:39Z")

</div>

Show us what you have in the function node please.

---

<div class="post-metadata">

**Author:** ![WhiteLion](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/whitelion/32/12266_2.png) [@WhiteLion](https://discourse.nodered.org/u/WhiteLion)\
**Post date:** [21 January 2020 11:19 UTC](https://discourse.nodered.org/t/file-transfers-fail-because-of-file-size-buffer-limit/20601/11 "2020-01-21T11:19:13Z")

</div>

this is the Next Cloud /WebDav (@kuehnelbs/node-red-contrib-nextcloud) flow:

```auto
[{"id":"36c9b7bc.f44008","type":"tab","label":"Flow 4","disabled":false,"info":""},{"id":"c7b1e5f8.6214f8","type":"nextcloud-webdav-out","z":"36c9b7bc.f44008","wname":"downloadFile","server":"ef9922e1.497078","filename":"","x":810,"y":580,"wires":[["ff51840d.3549f8"]]},{"id":"6f7b1091.8229d8","type":"nextcloud-webdav-list","z":"36c9b7bc.f44008","wname":"getNextcloudFiles","server":"ef9922e1.497078","directory":"/MusikBox/","x":830,"y":340,"wires":[["646b6028.7155a","7573f010.4db7e8"]]},{"id":"a73c022e.27a528","type":"function","z":"36c9b7bc.f44008","name":"dirExistsOnLocal","func":"// compares local Dir and nextcloud dir\n// Will check if a folder with the RfId exist on local...\n// if not so, then forwards the message, else sends no message\nlet ncFiles = msg.payload;\nlet localFiles = msg.localFiles;\n\nfor (let i in ncFiles) \n{\n //console.error(ncFiles[i]);\n //if(typeof(ncFiles[i].basename) === typeof 'undefined') continue;\n if(!ncFiles[i] || !ncFiles[i].basename || !ncFiles[i].basename.match(/\\.zip$/ig)) continue;\n msg.filename = ncFiles[i].filename;\n //msg.something = \"nothing\";\n \n let found = false;\n for (let x in localFiles) \n {\n if (localFiles[x] + \".zip\" === ncFiles[i].basename) \n { \n found = true; \n }\n }\n if(!found) \n { \n msg.download = ncFiles[i].basename;\n msg.parts = \"\";\n return [msg , null];\n } \n\n}\n\nreturn [null, msg];\n\n\n\n//for (i=0; i< len; i++) \n//{\n \n//}\n\n","outputs":2,"noerr":0,"x":830,"y":500,"wires":[["6079bd9d.333c7c","c7b1e5f8.6214f8"],["23f05d7b.e2016a"]]},{"id":"6079bd9d.333c7c","type":"debug","z":"36c9b7bc.f44008","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":1030,"y":440,"wires":[]},{"id":"646b6028.7155a","type":"fs-ops-dir","z":"36c9b7bc.f44008","name":"getLocalFiles","path":"/home/pi/Music/","pathType":"str","filter":"*","filterType":"str","dir":"localFiles","dirType":"msg","x":820,"y":400,"wires":[["a73c022e.27a528"]]},{"id":"f4521eda.42405","type":"debug","z":"36c9b7bc.f44008","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":1030,"y":620,"wires":[]},{"id":"23f05d7b.e2016a","type":"debug","z":"36c9b7bc.f44008","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":1030,"y":580,"wires":[]},{"id":"ced5b8ad.73915","type":"function","z":"36c9b7bc.f44008","name":"isResponseGood","func":"if(msg.responseUrl !== \"https://nc.xxxxxx.de/index.php/login\") return null;\nreturn msg;","outputs":1,"noerr":0,"x":830,"y":280,"wires":[["6f7b1091.8229d8"]]},{"id":"63cd4dba.b8e50c","type":"file","z":"36c9b7bc.f44008","name":"","filename":"","appendNewline":false,"createDir":true,"overwriteFile":"true","encoding":"none","x":830,"y":700,"wires":[[]]},{"id":"ff51840d.3549f8","type":"function","z":"36c9b7bc.f44008","name":"","func":"let targetDir = \"/home/pi/Music/test.zip\";\nmsg.filename = targetDir;\nreturn msg;","outputs":1,"noerr":0,"x":820,"y":640,"wires":[["63cd4dba.b8e50c"]]},{"id":"bddae6f9.780818","type":"inject","z":"36c9b7bc.f44008","name":"","topic":"","payload":"https://nc.xxxxxx.de","payloadType":"str","repeat":"","crontab":"","once":true,"onceDelay":"15","x":810,"y":200,"wires":[["b6c620df.de1c08","4fcf6cbc.435a5c"]]},{"id":"b6c620df.de1c08","type":"http request","z":"36c9b7bc.f44008","name":"","method":"GET","ret":"txt","paytoqs":false,"url":"https://nc.xxxxxx.de","tls":"","persist":false,"proxy":"","authType":"","x":810,"y":240,"wires":[["ced5b8ad.73915"]]},{"id":"7573f010.4db7e8","type":"debug","z":"36c9b7bc.f44008","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":1070,"y":360,"wires":[]},{"id":"4fcf6cbc.435a5c","type":"gc-trigger","z":"36c9b7bc.f44008","display":true,"name":"","x":990,"y":200,"wires":[[]]},{"id":"ef9922e1.497078","type":"nextcloud-credentials","z":"","cname":"XXXXX","address":"https://nc.xxxxxx.de","insecure":false}]

```

Also testing with FTP server node (node-red-contrib-ftp-server):

```auto
[{"id":"4e0dbbfd.623dfc","type":"ftp-server","z":"1edac7e4.ef6288","name":"xxxxxx","port":"2017","x":1080,"y":560,"wires":[["2b7cbed.aeac8c2"]]},{"id":"1d1db6f7.0a81b9","type":"file","z":"1edac7e4.ef6288","name":"","filename":"","appendNewline":false,"createDir":true,"overwriteFile":"true","encoding":"none","x":1530,"y":560,"wires":[[]]},{"id":"2b7cbed.aeac8c2","type":"function","z":"1edac7e4.ef6288","name":"","func":"let targetDir = \"/home/pi/Music/test123.zip\";\nmsg.filename = targetDir;\nreturn msg;","outputs":1,"noerr":0,"x":1290,"y":560,"wires":[["1d1db6f7.0a81b9"]]}]

```

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [21 January 2020 11:38 UTC](https://discourse.nodered.org/t/file-transfers-fail-because-of-file-size-buffer-limit/20601/12 "2020-01-21T11:38:16Z")

</div>

Does the webdav-out node fetch just one file or a whole directory of files?

---

<div class="post-metadata">

**Author:** ![WhiteLion](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/whitelion/32/12266_2.png) [@WhiteLion](https://discourse.nodered.org/u/WhiteLion)\
**Post date:** [21 January 2020 11:51 UTC](https://discourse.nodered.org/t/file-transfers-fail-because-of-file-size-buffer-limit/20601/13 "2020-01-21T11:51:48Z")

</div>

in my case only one file.... It can handle a list but I never tried

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [21 January 2020 12:35 UTC](https://discourse.nodered.org/t/file-transfers-fail-because-of-file-size-buffer-limit/20601/14 "2020-01-21T12:35:12Z")

</div>

Does the webdav node pass though other properties correctly ? If so can you attach the filename before calling it so you don't then need to do that afterwards... saving a copy etc...

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [21 January 2020 13:17 UTC](https://discourse.nodered.org/t/file-transfers-fail-because-of-file-size-buffer-limit/20601/15 "2020-01-21T13:17:34Z")

</div>

> [@dceejay](#):
>
> attach the filename before calling it so you don't then need to do that afterwards... saving a copy etc

Does adding a property to the message mean it has to be cloned?

---

<div class="post-metadata">

**Author:** ![WhiteLion](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/whitelion/32/12266_2.png) [@WhiteLion](https://discourse.nodered.org/u/WhiteLion)\
**Post date:** [21 January 2020 15:07 UTC](https://discourse.nodered.org/t/file-transfers-fail-because-of-file-size-buffer-limit/20601/16 "2020-01-21T15:07:12Z")

</div>

I agree with @Colin -\> Adding a property should not clone anything. - Does this lead to a copy of the object (including buffered file == out of memory) !? - Interessting question.

@dceejay I had this thougt too but dropped it because webdav itself uses the property msg.filename. So I cant declare it bevor.

The other bad thing is that 2GB swap seems to be unused.

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [21 January 2020 15:24 UTC](https://discourse.nodered.org/t/file-transfers-fail-because-of-file-size-buffer-limit/20601/17 "2020-01-21T15:24:34Z")

</div>

What does `top` show?

---

<div class="post-metadata">

**Author:** ![WhiteLion](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/whitelion/32/12266_2.png) [@WhiteLion](https://discourse.nodered.org/u/WhiteLion)\
**Post date:** [21 January 2020 15:41 UTC](https://discourse.nodered.org/t/file-transfers-fail-because-of-file-size-buffer-limit/20601/18 "2020-01-21T15:41:48Z")

</div>

[htop screeny](http://puu.sh/F2cyb/1706d038be.png)

EDIT: now sometimes webdav dl node comes up with mem allocation error:  
[click me](http://puu.sh/F2cEm/29faae639f.png)

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [21 January 2020 16:20 UTC](https://discourse.nodered.org/t/file-transfers-fail-because-of-file-size-buffer-limit/20601/19 "2020-01-21T16:20:22Z")

</div>

You are running mplayer and a linux desktop as well as Node-RED so I'm not at all surprised that you are running out of memory given that Node-RED has also taken 64% of RAM.

It would be more helpful though to sort the htop output by memory use than cpu time.

---

<div class="post-metadata">

**Author:** ![WhiteLion](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/whitelion/32/12266_2.png) [@WhiteLion](https://discourse.nodered.org/u/WhiteLion)\
**Post date:** [21 January 2020 16:45 UTC](https://discourse.nodered.org/t/file-transfers-fail-because-of-file-size-buffer-limit/20601/20 "2020-01-21T16:45:22Z")

</div>

I am running Mplayer because this is going to be a musicbox project. Killing it is no real scenario. I can reduce space by exiting some stuff like linux desktop...yes, but node red will eat 74% of the memory as you can see at second screen shot.  
It will not help freeing up 50 or 100 MB. When a bigger download is triggered after a fresh restart of node red it will eat the 600 MB free mem (I ve afer reboot). The garbage collector will only free up about 300 MB. (Mem leak I think) So the second bigger download will fail. - Doesnt matter if I free up some mem by exiting linux desktop. The problem remains: No chunked downloads possible... I am no expert at this but writing a datastream in small pakages is no new invention.... the file node maybe could do that with "append" option. ... while FTP and webdav nodes do not support this.

[Next page](https://discourse.nodered.org/t/file-transfers-fail-because-of-file-size-buffer-limit/20601.md?page=2)