# CSV ,Add extra column header to existing CSV file

**URL:** <https://discourse.nodered.org/t/csv-add-extra-column-header-to-existing-csv-file/12039>\
**Category:** General\
**Created:** [10 June 2019 14:54 UTC](https://discourse.nodered.org/t/csv-add-extra-column-header-to-existing-csv-file/12039 "2019-06-10T14:54:49Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Loi](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/loi/32/10736_2.png) [@Loi](https://discourse.nodered.org/u/Loi)\
**Post date:** [10 June 2019 14:54 UTC](https://discourse.nodered.org/t/csv-add-extra-column-header-to-existing-csv-file/12039/1 "2019-06-10T14:54:50Z")

</div>

Hi,

Im having a problem adding another header column to an existing CSV file.

Example,  
I have an existing CSV file with the following data

Header = Time, Temp , Humidity  
Data = 102236, 16, 45

This csv file constantly updated every minute and new data is appended to the csv file  
If I want to add an extra header to the existing csv file, how would i be able to write to it with out losing the data

So new CSV  
Header = Time,Temp,Humidity,Pressure  
Data = 102236,16,45,5

I appreciate your help  
Regards  
Loi

---

<div class="post-metadata">

**Author:** ![JayDickson](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/jaydickson/32/90592_2.png) [@JayDickson](https://discourse.nodered.org/u/JayDickson)\
**Post date:** [10 June 2019 15:00 UTC](https://discourse.nodered.org/t/csv-add-extra-column-header-to-existing-csv-file/12039/2 "2019-06-10T15:00:29Z")

</div>

You can't really add a column to CSV using an append, only rows. What you need to do is to read the CSV to memory (I'd probably go with an array of objects) add the column to the in-memory entity you've created, then write out to a new (or if you're feeling cocky, overwrite the existing) CSV.

---

<div class="post-metadata">

**Author:** ![Loi](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/loi/32/10736_2.png) [@Loi](https://discourse.nodered.org/u/Loi)\
**Post date:** [10 June 2019 15:02 UTC](https://discourse.nodered.org/t/csv-add-extra-column-header-to-existing-csv-file/12039/3 "2019-06-10T15:02:59Z")

</div>

Yes, that would work, but im afraid if the file gets too big it will crash node-red.  
the file would be over 100mb.

i will try it

Thanks Jay

---

<div class="post-metadata">

**Author:** ![JayDickson](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/jaydickson/32/90592_2.png) [@JayDickson](https://discourse.nodered.org/u/JayDickson)\
**Post date:** [10 June 2019 15:07 UTC](https://discourse.nodered.org/t/csv-add-extra-column-header-to-existing-csv-file/12039/4 "2019-06-10T15:07:49Z")

</div>

> [@Loi](#):
>
> the file would be over 100mb.

That is usually _way_ too big to be working in CSV format. Is there a technical reason that you've decided to go that route? I would highly recommend migrating something that large to some sort of database.

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [10 June 2019 21:49 UTC](https://discourse.nodered.org/t/csv-add-extra-column-header-to-existing-csv-file/12039/5 "2019-06-10T21:49:55Z")

</div>

What do want to do with the “old” data ? Add 0 in the new column or leave blank or ...

---

<div class="post-metadata">

**Author:** ![Loi](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/loi/32/10736_2.png) [@Loi](https://discourse.nodered.org/u/Loi)\
**Post date:** [11 June 2019 07:52 UTC](https://discourse.nodered.org/t/csv-add-extra-column-header-to-existing-csv-file/12039/6 "2019-06-11T07:52:06Z")

</div>

Hi dceejay,

I want to add a new header column, i want to keep the old data to have a look at the sensor data throughout the year. I will add more column throughout my project if needed.

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [11 June 2019 11:41 UTC](https://discourse.nodered.org/t/csv-add-extra-column-header-to-existing-csv-file/12039/7 "2019-06-11T11:41:40Z")

</div>

so You need to add some dummy value to the end of each just in case you add a second extra column ?

---

<div class="post-metadata">

**Author:** ![Loi](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/loi/32/10736_2.png) [@Loi](https://discourse.nodered.org/u/Loi)\
**Post date:** [12 June 2019 13:07 UTC](https://discourse.nodered.org/t/csv-add-extra-column-header-to-existing-csv-file/12039/8 "2019-06-12T13:07:11Z")

</div>

The new added column at the end of the original header will have new data.

---

<div class="post-metadata">

**Author:** ![Loi](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/loi/32/10736_2.png) [@Loi](https://discourse.nodered.org/u/Loi)\
**Post date:** [12 June 2019 13:10 UTC](https://discourse.nodered.org/t/csv-add-extra-column-header-to-existing-csv-file/12039/9 "2019-06-12T13:10:43Z")

</div>

The software im currently using to extract data uses csv and it compressing ratio is very good.

Jay, as mention above - you said i could read in the csv file and replace the header  
Im struggling with replacing the header with the new header.  
i can get the data and get the header of the file but i dont know how to get the new header with the old data i have.  
Please elabrote what node i could use.

Many thanks  
Loi

---

<div class="post-metadata">

**Author:** ![JayDickson](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/jaydickson/32/90592_2.png) [@JayDickson](https://discourse.nodered.org/u/JayDickson)\
**Post date:** [12 June 2019 14:00 UTC](https://discourse.nodered.org/t/csv-add-extra-column-header-to-existing-csv-file/12039/11 "2019-06-12T14:00:04Z")

</div>

> [@Loi](#):
>
> Please elabrote what node i could use.

Play with this. You'll need to edit the csv column template to match your schema, but this should work to update legacy CSVs with a new column.

`[{"id":"51cd58d4.63e4a8","type":"inject","z":"32697264.aa1c16","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":1040,"y":360,"wires":[["8114a6ea.1e9138"]]},{"id":"46d15bc3.978964","type":"csv","z":"32697264.aa1c16","name":"","sep":",","hdrin":true,"hdrout":"","multi":"mult","ret":"\\n","temp":"","skip":"0","x":1350,"y":360,"wires":[["c0b5860b.379e48"]]},{"id":"539b3829.709ed8","type":"change","z":"32697264.aa1c16","name":"Add column","rules":[{"t":"set","p":"payload.newcolumn","pt":"msg","to":"null","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":1270,"y":400,"wires":[["5f0fa5f7.f5475c"]]},{"id":"6456bc4d.498144","type":"csv","z":"32697264.aa1c16","name":"back to CSV","sep":",","hdrin":true,"hdrout":true,"multi":"one","ret":"\\n","temp":"","skip":"0","x":1050,"y":440,"wires":[["8ddb8c84.6a81"]]},{"id":"c0b5860b.379e48","type":"split","z":"32697264.aa1c16","name":"split (retaining order)","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":1080,"y":400,"wires":[["539b3829.709ed8"]]},{"id":"5f0fa5f7.f5475c","type":"join","z":"32697264.aa1c16","name":"merge array","mode":"auto","build":"string","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":1430,"y":400,"wires":[["6456bc4d.498144"]]},{"id":"8ddb8c84.6a81","type":"file","z":"32697264.aa1c16","name":"Destination file","filename":"","appendNewline":true,"createDir":false,"overwriteFile":"false","x":1220,"y":440,"wires":[[]]},{"id":"8114a6ea.1e9138","type":"file in","z":"32697264.aa1c16","name":"Source File","filename":"","format":"lines","chunk":false,"sendError":false,"x":1210,"y":360,"wires":[["46d15bc3.978964"]]}]`

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [12 June 2019 14:33 UTC](https://discourse.nodered.org/t/csv-add-extra-column-header-to-existing-csv-file/12039/12 "2019-06-12T14:33:17Z")

</div>

> [@Loi](#):
>
> i want to keep the old data to have a look at the sensor data throughout the year

You need to import the data to a database. Then you can have additional fields as needed. SQLite would be a good place to start as it doesn't need an additional service running.

I would try to take a copy of the CSV periodically - though as you are writing it so often, that may be a challenge as you might find the end of the file being caught mid-write. There are probably several ways to mitigate that depending on the details of your sensor and whether it matters if you were to occasionally lose a reading. Clearing down the CSV file would help keep things manageable.

Of course, if you can , you should change the source to write to something that Node-RED can consume directly. Possibly you could substitute a UNIX Pipe instead of the CSV file for example.

---

<div class="post-metadata">

**Author:** ![zenofmud](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/zenofmud/32/316_2.png) [@zenofmud](https://discourse.nodered.org/u/zenofmud)\
**Post date:** [12 June 2019 16:10 UTC](https://discourse.nodered.org/t/csv-add-extra-column-header-to-existing-csv-file/12039/13 "2019-06-12T16:10:16Z")

</div>

> [@Loi](#):
>
> The new added column at the end of the original header will have new data.

I think what Dave wad hinting at is the question "When you add a new column, what will be in that column for all the rows that previously existed?"

You need to keep that in mind.

---

<div class="post-metadata">

**Author:** ![Loi](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/loi/32/10736_2.png) [@Loi](https://discourse.nodered.org/u/Loi)\
**Post date:** [13 June 2019 13:13 UTC](https://discourse.nodered.org/t/csv-add-extra-column-header-to-existing-csv-file/12039/14 "2019-06-13T13:13:06Z")

</div>

I will be storing the data to a cloud database periodically. Im very grateful for the information and will consider it once i have this up and running ; )

---

<div class="post-metadata">

**Author:** ![Loi](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/loi/32/10736_2.png) [@Loi](https://discourse.nodered.org/u/Loi)\
**Post date:** [13 June 2019 13:16 UTC](https://discourse.nodered.org/t/csv-add-extra-column-header-to-existing-csv-file/12039/15 "2019-06-13T13:16:10Z")

</div>

Sorry, I misunderstand Dave questions. As the data was not there for the new column all the the row for that column will be blank.  
Im hoping it will write the new value for that column on the same row as the other reading.

---

<div class="post-metadata">

**Author:** ![Loi](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/loi/32/10736_2.png) [@Loi](https://discourse.nodered.org/u/Loi)\
**Post date:** [13 June 2019 13:19 UTC](https://discourse.nodered.org/t/csv-add-extra-column-header-to-existing-csv-file/12039/16 "2019-06-13T13:19:30Z")

</div>

Thanks Jay.

I have been working on it and im having trouble where after i Split it, then add in the new header and then join them again and the transfer to the 'CSV' node it only output the header string and doesnt send the data that came with that i read in from the file.  
Even though i can see all the arrays in the join node

[flows (5).json (2.4 KB)](https://discourse.nodered.org/uploads/short-url/6bRcBXdDkoONERouHUI8ItPqRZF.json)

---

<div class="post-metadata">

**Author:** ![JayDickson](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/jaydickson/32/90592_2.png) [@JayDickson](https://discourse.nodered.org/u/JayDickson)\
**Post date:** [13 June 2019 13:58 UTC](https://discourse.nodered.org/t/csv-add-extra-column-header-to-existing-csv-file/12039/17 "2019-06-13T13:58:39Z")

</div>

> [@JayDickson](#):
>
> You'll need to edit the csv column template to match your schema

 ![17%20AM](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/2X/5/58f0d521e9b684e5b302be04672459dd421e66e8.png)

Use your own column names of course. Can't show you what my output looks like because it has PHI in it, but hopefully this gets you on the right track.

---

<div class="post-metadata">

**Author:** ![Loi](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/loi/32/10736_2.png) [@Loi](https://discourse.nodered.org/u/Loi)\
**Post date:** [13 June 2019 14:01 UTC](https://discourse.nodered.org/t/csv-add-extra-column-header-to-existing-csv-file/12039/18 "2019-06-13T14:01:44Z")

</div>

is it possible to send in a "msg." for the column template instead of editing inside the csv node?

---

<div class="post-metadata">

**Author:** ![Loi](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/loi/32/10736_2.png) [@Loi](https://discourse.nodered.org/u/Loi)\
**Post date:** [13 June 2019 14:04 UTC](https://discourse.nodered.org/t/csv-add-extra-column-header-to-existing-csv-file/12039/19 "2019-06-13T14:04:01Z")

</div>

is it possible to send in a "msg." for the column template instead of editing inside the csv node?

---

<div class="post-metadata">

**Author:** ![JayDickson](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/jaydickson/32/90592_2.png) [@JayDickson](https://discourse.nodered.org/u/JayDickson)\
**Post date:** [13 June 2019 14:04 UTC](https://discourse.nodered.org/t/csv-add-extra-column-header-to-existing-csv-file/12039/20 "2019-06-13T14:04:12Z")

</div>

Not to my knowledge, though this operation should only really be done if you change your schema, which shouldn't be a frequent event.

---

<div class="post-metadata">

**Author:** ![zenofmud](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/zenofmud/32/316_2.png) [@zenofmud](https://discourse.nodered.org/u/zenofmud)\
**Post date:** [13 June 2019 14:04 UTC](https://discourse.nodered.org/t/csv-add-extra-column-header-to-existing-csv-file/12039/21 "2019-06-13T14:04:47Z")

</div>

> [@Loi](#):
>
> Sorry, I misunderstand Dave questions. As the data was not there for the new column all the the row for that column will be blank.

I hope you take that into consideration if you ever try to use the new column in calculations that include the old rows....

[Next page](https://discourse.nodered.org/t/csv-add-extra-column-header-to-existing-csv-file/12039.md?page=2)