I'm trying to traverse through a large CSV file. Each line, I do some data remodelling and convert it to XML format. Lastly, I save all of it into an XML file. In my actual implementation, the resulting remodelled XML data is 10x larger than the source CSV data. I have a use case where the sourc…

So I tried appending the XML data per iteration and delete the msg.payload before feeding the message to the Join node so it doesn't accumulate data. If you want to iterate, you need to use output msg per line in the file-in node. Still going to be heavy impact. I don't think node-red is the ri…

Yes. Using msg per line on the input should allow this to work. You shouldn’t need the join hopefully ax you can also output a line at a time as well. I have been processing video files extracting metadata no problem.

[image] bakman2: I don't think node-red is the right tool to perform these action on such a large csv, a csv that size should belong in a database where you can do all kinds of operations on the data and iterate more efficiently (if the data is indexed) Yes, I agree with that. But for some r…

When you use the file in node in per line mode it add a msg.parts property that includes the number of the line - and when complete, the total number of lines - so you could possibly use those a) to trigger adding the header - and then b) adding the tailer.

Node-RED

Node-RED Forum

Memory issues processing and writing a large file

General

Topic		Replies	Views
Memory issue processing large csv file General	2	778	21 October 2021
Read large JSON file (to insert to MySQL) causes Node-RED crash due to JavaScript heap out of memory General	10	1888	21 September 2020
Node red crashes after ~1h "heap out of memory" General	33	501	10 August 2025
File transfers fail because of file size - buffer limit? General	23	2137	21 January 2020
A new type of flow General	38	655	26 September 2024

Copyright OpenJS Foundation and Node-RED contributors. All rights reserved. The OpenJS Foundation has registered trademarks and uses trademarks. For a list of trademarks of the OpenJS Foundation, please see our Trademark Policy and Trademark List. Trademarks and logos not indicated on the list of OpenJS Foundation trademarks are trademarks™ or registered® trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.

Memory issues processing and writing a large file

Related topics