# Log data into text file

**URL:** <https://discourse.nodered.org/t/log-data-into-text-file/42716>\
**Category:** General\
**Created:** [16 March 2021 13:36 UTC](https://discourse.nodered.org/t/log-data-into-text-file/42716 "2021-03-16T13:36:07Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![sathishkumarc14](https://avatars.discourse-cdn.com/v4/letter/s/e480ec/32.png) [@sathishkumarc14](https://discourse.nodered.org/u/sathishkumarc14)\
**Post date:** [16 March 2021 13:36 UTC](https://discourse.nodered.org/t/log-data-into-text-file/42716/1 "2021-03-16T13:36:07Z")

</div>

I am having a scenario as follows,

1. TCP listener node receives a message and storing at text (each line have the payload). And it is connected with 5000 client connections and receiving data with every 1 minute.

```auto
[{"id":"9256fb27.499c88","type":"tcp in","z":"319467b8.0c7a48","name":"","server":"server","host":"","port":"1000","datamode":"stream","datatype":"utf8","newline":"","topic":"","base64":false,"x":100,"y":240,"wires":[["746239af.cd87b8"]]},{"id":"746239af.cd87b8","type":"simple-queue","z":"319467b8.0c7a48","name":"Qu","firstMessageBypass":true,"bypassInterval":"0","x":282,"y":240,"wires":[["ff655832.a46e18"]]},{"id":"ff655832.a46e18","type":"function","z":"319467b8.0c7a48","name":"Raw Data","func":"\nvar currentDateString = dateFormat();\n\n/ **Define file name** /\nvar fileName = \"E:\\\\CPA\\\\Data\\\\\" + currentDateString + \".txt\";\nmsg.filename = fileName;\n\nvar payload = {\"payload\" : msg.payload, \"ip\" : msg.ip};\nmsg.payload = payload;\n\nreturn msg;\n\n\n/ **date string function** /\nfunction dateFormat()\n{\n var cDate = new Date();\n var dateString = (cDate.getDate() < 10) ? (\"0\" + cDate.getDate()) : cDate.getDate();\n var monthString = ((cDate.getMonth() + 1) < 10) ? (\"0\" + (cDate.getMonth() + 1)) : (cDate.getMonth() + 1);\n var yearString = cDate.getFullYear();\n \n var fullDate = dateString + \"-\" + monthString + \"-\" + yearString;\n return fullDate;\n}\n ","outputs":1,"noerr":0,"initialize":"","finalize":"","x":440,"y":240,"wires":[["8e415a0a.febaf8"]]},{"id":"8e415a0a.febaf8","type":"file","z":"319467b8.0c7a48","name":"Write as data","filename":"","appendNewline":true,"createDir":true,"overwriteFile":"false","encoding":"none","x":610,"y":240,"wires":[["6ab46437.b6090c"]]},{"id":"6ab46437.b6090c","type":"function","z":"319467b8.0c7a48","name":"Inject","func":"msg = {};\nmsg.trigger = 1;\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":450,"y":320,"wires":[["746239af.cd87b8"]]}]

```

1. Now we read reading (file in) the each line of a data as payload and trying to send to Azure IoT Hub.

Text file logging is getting slow after reaching the size of 50 to 60 MB. Is there any other methods to handle such a large files?  
What about concurrent access of the text file? (Both read and write)

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [16 March 2021 13:41 UTC](https://discourse.nodered.org/t/log-data-into-text-file/42716/2 "2021-03-16T13:41:43Z")

</div>

> [@sathishkumarc14](#):
>
> Text file logging is getting slow after reaching the size of 50 to 60 MB. Is there any other methods to handle such a large files?

Start a new file every hour?  
Send directly to Azure?  
Use a database?

---

<div class="post-metadata">

**Author:** ![sathishkumarc14](https://avatars.discourse-cdn.com/v4/letter/s/e480ec/32.png) [@sathishkumarc14](https://discourse.nodered.org/u/sathishkumarc14)\
**Post date:** [17 March 2021 05:18 UTC](https://discourse.nodered.org/t/log-data-into-text-file/42716/3 "2021-03-17T05:18:41Z")

</div>

1. Now we are trying to store one day data in single file. Anyways, starting the new file for each hour is good idea.
2. Yes, reading each line and send to azure IoT hub with REST.
3. We are not using any database. Is that right idea to use a data base?

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [17 March 2021 10:12 UTC](https://discourse.nodered.org/t/log-data-into-text-file/42716/4 "2021-03-17T10:12:52Z")

</div>

> [@sathishkumarc14](#):
>
> Yes, reading each line and send to azure IoT hub with REST

I meant, don't bother with a file - just send data directly to aws

> [@sathishkumarc14](#):
>
> We are not using any database. Is that right idea to use a data base?

Depends if you want to keep a local copy of data. If yes, then for the amount you are speaking about, probably worth setting up a database.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [16 May 2021 10:13 UTC](https://discourse.nodered.org/t/log-data-into-text-file/42716/5 "2021-05-16T10:13:47Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.