Log data into text file

General
1

I am having a scenario as follows,

  1. TCP listener node receives a message and storing at text (each line have the payload). And it is connected with 5000 client connections and receiving data with every 1 minute.
[{"id":"9256fb27.499c88","type":"tcp in","z":"319467b8.0c7a48","name":"","server":"server","host":"","port":"1000","datamode":"stream","datatype":"utf8","newline":"","topic":"","base64":false,"x":100,"y":240,"wires":[["746239af.cd87b8"]]},{"id":"746239af.cd87b8","type":"simple-queue","z":"319467b8.0c7a48","name":"Qu","firstMessageBypass":true,"bypassInterval":"0","x":282,"y":240,"wires":[["ff655832.a46e18"]]},{"id":"ff655832.a46e18","type":"function","z":"319467b8.0c7a48","name":"Raw Data","func":"\nvar currentDateString = dateFormat();\n\n/** Define file name **/\nvar fileName = \"E:\\\\CPA\\\\Data\\\\\" + currentDateString + \".txt\";\nmsg.filename = fileName;\n\nvar payload = {\"payload\" : msg.payload, \"ip\" : msg.ip};\nmsg.payload = payload;\n\nreturn msg;\n\n\n/** date string function **/\nfunction dateFormat()\n{\n    var cDate = new Date();\n    var dateString = (cDate.getDate() < 10) ? (\"0\" + cDate.getDate()) : cDate.getDate();\n    var monthString = ((cDate.getMonth() + 1) < 10) ? (\"0\" + (cDate.getMonth() + 1)) : (cDate.getMonth() + 1);\n    var yearString = cDate.getFullYear();\n    \n    var fullDate = dateString + \"-\" + monthString + \"-\" + yearString;\n    return fullDate;\n}\n    ","outputs":1,"noerr":0,"initialize":"","finalize":"","x":440,"y":240,"wires":[["8e415a0a.febaf8"]]},{"id":"8e415a0a.febaf8","type":"file","z":"319467b8.0c7a48","name":"Write as data","filename":"","appendNewline":true,"createDir":true,"overwriteFile":"false","encoding":"none","x":610,"y":240,"wires":[["6ab46437.b6090c"]]},{"id":"6ab46437.b6090c","type":"function","z":"319467b8.0c7a48","name":"Inject","func":"msg = {};\nmsg.trigger = 1;\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":450,"y":320,"wires":[["746239af.cd87b8"]]}]
  1. Now we read reading (file in) the each line of a data as payload and trying to send to Azure IoT Hub.

Text file logging is getting slow after reaching the size of 50 to 60 MB. Is there any other methods to handle such a large files?
What about concurrent access of the text file? (Both read and write)

2

Start a new file every hour?
Send directly to Azure?
Use a database?

3
  1. Now we are trying to store one day data in single file. Anyways, starting the new file for each hour is good idea.
  2. Yes, reading each line and send to azure IoT hub with REST.
  3. We are not using any database. Is that right idea to use a data base?
4

I meant, don't bother with a file - just send data directly to aws

Depends if you want to keep a local copy of data. If yes, then for the amount you are speaking about, probably worth setting up a database.

5

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.