Reluctantly asking for help how to handle huge data files

General
1

Maybe because the "log files" are Node-Red generated I should ask here.

For the past 18 months - or there about - I have been logging the temperature every.... 30 seconds.

The file is text and is time stamped.

So an extract of the file looks like:

2021-04-30 23:55:14 22.00 Deg C 22.00 Deg C
2021-04-30 23:57:22 21.00 Deg C 21.00 Deg C
2021-04-30 23:59:27 21.00 Deg C 22.00 Deg C
2021-05-01 00:01:30 21.00 Deg C 21.00 Deg C
2021-05-01 00:03:35 22.00 Deg C 22.00 Deg C
2021-05-01 00:05:40 22.00 Deg C 22.00 Deg C
2021-05-01 00:07:40 21.00 Deg C 21.00 Deg C
2021-05-01 00:09:51 21.00 Deg C 21.00 Deg C

(I now see I messed up with something, but that's by the by in the scheme of things)

So you have a date; time and temperature.
And ok, they are happening every 5 seconds.

So I want to handle these files and draw graphs with "averages" depending on how much you "zoom in".

Or the other way when you "zoom out".
So (going with the first one) you would see averages for the week - for the month. So you would see 4 weeks.
Zoom in and it goes to days. You would see the days.
Zoom in again and you would see the 24 hours for the day.
Zoom in again and you see the minutes
Zoom in again and you see the seconds.

At each level the averages are shown. "Averages"... Not sure if that is the correct word.
There are: averages; means; and other terms I don't know.

This is not something I do, so although I have the idea of what I want to do: actually getting it happening is a whole other kettle of fish for me.

Thanks folks.

2

I think you can use InfluxDB + Grafana to achieve that

2 Likes
3

Is that really necessary?

4

It is actually every 5 seconds... But who's counting?

That is because I want/have a real time display of the temperature.

I just stuck the logging at the end.

I have no experience with InfluxDB and/or grafana....

Seems like I am in for some learning.

5

Actually, you can never have that due to the speed of light :wink:

1 Like
6

Way back when, I just got the sensor to spit out the reading every 5 seconds.
It is displayed on an LED screen and ...

anyway.

I may have to learn a language to parse the file and generate the needed files for the time span required.

That may be easier. It isn't I need all the time access to the data. I just have the log and want to see if there are any trends that may need further research to what is happening.

7

As @mbonani said, InfluxDB and Grafana are ideal for this sort of thing. It's what I, and a lot of others here, use for this sort of thing.

1 Like
8

Thanks.

I shall have to dive in to that world and learn about it.

9

I'm surprised that you are only measuring the temperature to the nearest integer :wink:

4 Likes
10

From the data you posted, the interval is every two minutes instead of 30 seconds?

Yes InfluxDB plus Grafana will do the job well.

11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.