I am working on Datalogger using Node red programming tool.
Read Data from serial node -> Store parameters in a csv formatted file.
Read data from Csv formatted file and Push data to the cloud using MQTT Node.
Here, i am following blob like folder structure to store data in a csv file(Year/Month/Day/Filename.csv).
For example: 2020/July/12/2020July12.csv.
Data stored in the file per day is minimum 512 KB.
Now i am successfully completed the Process1 by storing the data in the file the way i wanted.
But when it comes to reading, file-in node read the complete csv file and load into memory and send line by line in a single shot.
Assume am storing data every 30 seconds and am sending 5 records every one hour.
Here the storing and retrieving process is completely independent.
Sometimes due to poor network, i may not be able to push data to the server for 3, 4 days.
I can't use Watch node / delay node (rate limiting) for my scenario
i need a node that should read a file line by line for every inject.