I am working on Datalogger using Node red programming tool.
Project scenario:
Process 1:
Read Data from serial node -> Store parameters in a csv formatted file.
Process 2:
Read data from Csv formatted file and Push data to the cloud using MQTT Node.
Here, i am following blob like folder structure to store data in a csv file(Year/Month/Day/Filename.csv).
For example: 2020/July/12/2020July12.csv.
Data stored in the file per day is minimum 512 KB.
Now i am successfully completed the Process1 by storing the data in the file the way i wanted.
But when it comes to reading, file-in node read the complete csv file and load into memory and send line by line in a single shot.
Assume am storing data every 30 seconds and am sending 5 records every one hour.
Here the storing and retrieving process is completely independent.
Sometimes due to poor network, i may not be able to push data to the server for 3, 4 days.
I can't use Watch node / delay node (rate limiting) for my scenario
i need a node that should read a file line by line for every inject.
That doesn't work in that way. In a single inject it read complete file and send all the lines to the debug node.
Is that the efficient way to read whole file into RAM and process the records?
In python, we are able to read a specific number of lines from a text file.
Assume I have 100 lines of data in a file.
I Inject flow every 1 minute.
In my first inject, i read first 5 lines(1,2,3,4 and 5) and publish the data to the server and wait for acknowledgement.
If getting, the ack, I will store the line number as 6 (for Next inject i have to start read from line number 6)in a context memory.
In my second inject, I read another 5 lines starting from the 6th line up to 10th line and publish the data to the server and wait for acknowledgement.
If not, getting the acknowledgement, then i will reload the same lines of data on my thrid inject. And i keep update the line number am currently reading(i.e. Read pointer) in a Context memory.
If this is the scenario, for every inject it loads whole file.
What will be the result, if the file size is too large.
I am looking forward to hear your favourable advices.
so - the file-in read by line mode does work in streaming mode so it shouldn't read the whole file into memory in one go - but yes does only read the whole file at a time. So yes you could do something similar to what you want but would indeed be re-reading the whole file each time. You may be better using the tail node that does a file tail function and only sends the last added lines as they are added. You could then just hold that relatively few lines in local context memory until you have sent them - or maybe use one of the queue nodes to hold them until processed.
I have tried the above method to filter the specific lines from a file with the help of index property using switch node.
As u said, this would be re-reading the whole file each time.
Instead, let me try to solve this scenario with the help of Queue node.
Thing am going to do..
Assume i am having 1000 lines of code and i am going to send 5 lines for every 1 minute.
In my first inject, i will load first 100 lines of data into queue with the help of switch node to filter the index(this time 0 to 100). And there will be only 100 records in a queue.
After the queue becomes empty, again i will load another 100 lines of data with the help of switch node to filter the index**(this time 101 to 200)**. And i will repeat the same process for the rest.
The way I would do this is to use a FIFO buffer in file backed global context memory to buffer the data prior to sending to your server. No need to mess about formatting, writing, and reading files.
Hey Paul, while what you are saying is certainly a very reasonable workaround (and will likely satisfy 99% of requirements), it doesn't 100% confirm it was your mqtt pub (or your node-red msg) that published it. (not TRUE QoS)
If you want true QoS (in the MQTT definition of QoS), the best answer would be for the client lib making the pub call to return to the caller after the PUBACK.
And anyhow, theres nothing quite as good or useful as attaching a node to an output (in a serial wiring fashion) to guarantee (in node-red terms) it was your message travelling down the wire (so any extra stuff you have in the msg object would get passed through too).
Status is generally meant to indicate things like the state of the connection not individual messages especially when it can handle 000s of msgs a second. ( plus the editor is not a dashboard etc)
You can use a Complete node linked to the MQTT node to show when it has completed action on each message. A quick test suggests that it doesn't signal Complete if it is unable to send the message to the broker.
