Would not the actual write operation be handled by a different process in the OS?
My understanding is -
I/O gets offloaded to worker threads and main thread continues .
Anyways i will check which node causes the issue.
Hi -
Update.
Found the issue. As mentioned earlier - my flow
- Gets data from DB
- Checks if payload is empty (switch node)
- Converts to CSV ( csv node)
- replaces some characters using regexp (function node)
- To file node.
The culprit which blocking the NR and the editor everytime a huge chunk of records (100K --> 31MB) worth of data - came was the csv node.
I replaced the node with a function node and now even 200K is written in a jiffy - without blocking NR - editor / runtime.
I am aware there will be an upper ceiling before things crack but at least its not 100K or such .
Thank You Everyone for pointing me in the correct direction.
Which node did you replace. What did you do instead?
Details please so we can (if necessary) fix things.
Sure.
My "csv" node - has below per my need 
Replaced it with function node - with below code :
I am sure this can easy be made generic.
var output='';
if (msg.csvSeperator == undefined) {
msg.csvSeperator = ',';
}
if (msg.sendCsvHeaders == undefined) {
msg.sendCsvHeaders = true;
}
if(Array.isArray(msg.payload) && msg.payload.length >0 ) {
if(msg.sendCsvHeaders){
output = Object.keys(msg.payload[0]);
output = output.join(msg.csvSeperator) + "\n";
}
for(let cnt=0;cnt<msg.payload.length;cnt++)
{output = output+ Object.values(msg.payload[cnt]).join(msg.csvSeperator) + "\n";
}
msg.payload=output;
} else{
msg.payload = '';
}
return msg;
I am aware this will have top ceiling before it causes main thread to be blocked - but for that i ensure the load never goes beyond a threshold.
That code is not very robust. If you ever end up with output containing some odd characters (most notably a comma), it will fail to produce the output you expect.
Also, you should be doing any text manipulation with the original object not the CSV object since the original will be much faster to manipulate.
Input to make it robust would be welcome 
Use a tried and tested CSV lib.
IMO, the lib used in the CSV node is not great
- Really REALLY slow (in my test data of 20000 rows, most libs take < 100ms. The built in CSV takes an excruciating 8sec)
- It is Not RFC4180 compliant (see here)
(something I hope to get resolved for NR 4.0.0)
A lib I have used that is both faster AND mostly RFC4180 compliant is called comma-separated-values
This is how you use it - 1 line of code.
---
To backup @TotallyInformation, here is why your code is "not great" - it fails every RFC test I keep handy for checking out CSV libs:
1 Like
Thanks Steve. Will check. Getting new libraries for our prod env may get tricky.
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.
