# Using AWK on CSV file using function node

**URL:** <https://discourse.nodered.org/t/using-awk-on-csv-file-using-function-node/47251>\
**Category:** General\
**Created:** [16 June 2021 14:57 UTC](https://discourse.nodered.org/t/using-awk-on-csv-file-using-function-node/47251 "2021-06-16T14:57:55Z")\
**Posts on this page:** 1\
**Showing post:** 5

<div class="post-metadata">

**Author:** ![sahsha](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/sahsha/32/43845_2.png) [@sahsha](https://discourse.nodered.org/u/sahsha)\
**Post date:** [17 June 2021 06:39 UTC](https://discourse.nodered.org/t/using-awk-on-csv-file-using-function-node/47251/5 "2021-06-17T06:39:57Z")

</div>

Thankyou. Didn't thought it to be this simple 😬

there is one more thing with this flow. I also need to remove the duplicate values. I was going through solutions given below. How can I make them use in this flow.

> [@Ui\_dropdown: set msg.options for multiple entries](https://discourse.nodered.org/t/ui-dropdown-set-msg-options-for-multiple-entries/37318/2):
>
> Your function is not returning the msg. so no option set. Then your change node overwrites msg.option with msg.payload. But to only pass on changed msg.options compare the array lengths. e.g. [{"id":"f454e332.520a5","type":"inject","z":"63af3844.ce5a8","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"test1/Node1/2/Data","payload":"{\"ID\":\"13dfsl48\",\"TimeStamp\":0,\"SensorNamwe\":\"Sensor\_3\",\"SensorRaw\":-332,\"Sen…

> [@Remove duplicates from an array of objects](https://discourse.nodered.org/t/remove-duplicates-from-an-array-of-objects/28013/2):
>
> I would take the lazy approach and use a function node. var prev = null; msg.payload = msg.payload.filter((item) =\> { var keep = (item.x !== prev); prev = item.x; return keep; }); return msg; This assumes you only want to dedupe on 'x', and they are all in sequence. You would have to modify it a bit if that's not the case.

---

_[View the full topic](https://discourse.nodered.org/t/using-awk-on-csv-file-using-function-node/47251)._