I'm trying to traverse through a large CSV file. Each line, I do some data remodelling and convert it to XML format. Lastly, I save all of it into an XML file. In my actual implementation, the resulting remodelled XML data is 10x larger than the source CSV data. I have a use case where the sourc…

So I tried appending the XML data per iteration and delete the msg.payload before feeding the message to the Join node so it doesn't accumulate data. If you want to iterate, you need to use output msg per line in the file-in node. Still going to be heavy impact. I don't think node-red is the ri…

Memory issues processing and writing a large file

General

zenofmud Split this topic 5 September 2021 11:20 6

A post was split to a new topic: Converting CSV to mysql causes error with blank csv columns

Converting CSV to mysql causes error with blank csv columns

Topic		Replies	Views
Memory issue processing large csv file General	3	756	20 December 2021
Read large JSON file (to insert to MySQL) causes Node-RED crash due to JavaScript heap out of memory General	11	1771	5 October 2020
File transfers fail because of file size - buffer limit? General	23	1982	21 January 2020
A new type of flow General	39	440	25 December 2024
File writing blocks NR? General	29	380	16 August 2023

Memory issues processing and writing a large file

Related topics