# File writing blocks NR?

**URL:** <https://discourse.nodered.org/t/file-writing-blocks-nr/79021>\
**Category:** General\
**Created:** [7 June 2023 13:30 UTC](https://discourse.nodered.org/t/file-writing-blocks-nr/79021 "2023-06-07T13:30:10Z")\
**Posts on this page:** 10\
**Page:** 2

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [8 June 2023 12:43 UTC](https://discourse.nodered.org/t/file-writing-blocks-nr/79021/21 "2023-06-08T12:43:00Z")

</div>

Would not the actual write operation be handled by a different process in the OS?

---

<div class="post-metadata">

**Author:** ![SandeepA](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/sandeepa/32/63225_2.png) [@SandeepA](https://discourse.nodered.org/u/SandeepA)\
**Post date:** [8 June 2023 17:30 UTC](https://discourse.nodered.org/t/file-writing-blocks-nr/79021/22 "2023-06-08T17:30:39Z")

</div>

> [@TotallyInformation](#):
>
> Again, there is no other thread. Node.js is single threaded.

My understanding is -  
I/O gets offloaded to worker threads and main thread continues .  
Anyways i will check which node causes the issue.

---

<div class="post-metadata">

**Author:** ![SandeepA](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/sandeepa/32/63225_2.png) [@SandeepA](https://discourse.nodered.org/u/SandeepA)\
**Post date:** [12 June 2023 16:28 UTC](https://discourse.nodered.org/t/file-writing-blocks-nr/79021/23 "2023-06-12T16:28:19Z")

</div>

Hi -

Update.

Found the issue. As mentioned earlier - my flow

1. Gets data from DB
2. Checks if payload is empty (switch node)
3. Converts to CSV ( csv node)
4. replaces some characters using regexp (function node)
5. To file node.

The culprit which blocking the NR and the editor everytime a huge chunk of records (100K --\> 31MB) worth of data - came was the csv node.

I replaced the node with a function node and now even 200K is written in a jiffy - without blocking NR - editor / runtime.

I am aware there will be an upper ceiling before things crack but at least its not 100K or such .

Thank You Everyone for pointing me in the correct direction.

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [12 June 2023 17:11 UTC](https://discourse.nodered.org/t/file-writing-blocks-nr/79021/24 "2023-06-12T17:11:58Z")

</div>

> [@SandeepA](#):
>
> I replaced the node with a function node and now even 200K is written in a jiffy

Which node did you replace. What did you do instead?

Details please so we can (if necessary) fix things.

---

<div class="post-metadata">

**Author:** ![SandeepA](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/sandeepa/32/63225_2.png) [@SandeepA](https://discourse.nodered.org/u/SandeepA)\
**Post date:** [12 June 2023 18:19 UTC](https://discourse.nodered.org/t/file-writing-blocks-nr/79021/25 "2023-06-12T18:19:30Z")

</div>

Sure.

My "csv" node - has below per my need 👎  
 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/7/9/798204d60afd949a634c3f95ccbf828f7ac25727.png)

Replaced it with function node - with below code :  
I am sure this can easy be made generic.

```auto
var output='';

if (msg.csvSeperator == undefined) {
msg.csvSeperator = ',';
}

if (msg.sendCsvHeaders == undefined) {
msg.sendCsvHeaders = true;
}

if(Array.isArray(msg.payload) && msg.payload.length >0 ) {
  if(msg.sendCsvHeaders){
   output = Object.keys(msg.payload[0]);
   output = output.join(msg.csvSeperator) + "\n";
   }
   
   for(let cnt=0;cnt<msg.payload.length;cnt++)
   {output = output+ Object.values(msg.payload[cnt]).join(msg.csvSeperator) + "\n";
   }
msg.payload=output;
    
} else{
   msg.payload = ''; 
}

return msg;

```

I am aware this will have top ceiling before it causes main thread to be blocked - but for that i ensure the load never goes beyond a threshold.

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [13 June 2023 16:50 UTC](https://discourse.nodered.org/t/file-writing-blocks-nr/79021/26 "2023-06-13T16:50:31Z")

</div>

That code is not very robust. If you ever end up with output containing some odd characters (most notably a comma), it will fail to produce the output you expect.

Also, you should be doing any text manipulation with the original object not the CSV object since the original will be much faster to manipulate.

---

<div class="post-metadata">

**Author:** ![SandeepA](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/sandeepa/32/63225_2.png) [@SandeepA](https://discourse.nodered.org/u/SandeepA)\
**Post date:** [15 June 2023 08:06 UTC](https://discourse.nodered.org/t/file-writing-blocks-nr/79021/27 "2023-06-15T08:06:44Z")

</div>

> [@TotallyInformation](#):
>
> That code is not very robust. If you ever end up with output containing some odd characters (most notably a comma), it will fail to produce the output you expect.

Input to make it robust would be welcome 🙂

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [15 June 2023 12:05 UTC](https://discourse.nodered.org/t/file-writing-blocks-nr/79021/28 "2023-06-15T12:05:03Z")

</div>

> [@SandeepA](#):
>
> Input to make it robust would be welcome

Use a tried and tested CSV lib.

IMO, the lib used in the CSV node is not great

- Really REALLY slow (in my test data of 20000 rows, most libs take \< 100ms. The built in CSV takes an excruciating 8sec)
- It is Not RFC4180 compliant (see [here](https://github.com/node-red/node-red/issues/3934))

_(something I hope to get resolved for NR 4.0.0)_

A lib I have used that is both faster AND _mostly_ RFC4180 compliant is called **`comma-separated-values`**

This is how you use it - 1 line of code.

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/2/5/252b1edc5221349eda6ada9ec50b78baf4491454.png)

  
---

To backup @TotallyInformation, here is why your code is "not great" - it fails every RFC test I keep handy for checking out CSV libs:

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/7/f/7ffddfb6c42e6ddeeab2834dc2101a5d35886b60.png)

---

<div class="post-metadata">

**Author:** ![SandeepA](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/sandeepa/32/63225_2.png) [@SandeepA](https://discourse.nodered.org/u/SandeepA)\
**Post date:** [17 June 2023 10:33 UTC](https://discourse.nodered.org/t/file-writing-blocks-nr/79021/29 "2023-06-17T10:33:15Z")

</div>

Thanks Steve. Will check. Getting new libraries for our prod env may get tricky.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [16 August 2023 10:33 UTC](https://discourse.nodered.org/t/file-writing-blocks-nr/79021/30 "2023-08-16T10:33:29Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.

[Previous page](https://discourse.nodered.org/t/file-writing-blocks-nr/79021.md?page=1)