# Reluctantly asking for help how to handle huge data files

**URL:** <https://discourse.nodered.org/t/reluctantly-asking-for-help-how-to-handle-huge-data-files/48959>\
**Category:** General\
**Created:** [27 July 2021 09:54 UTC](https://discourse.nodered.org/t/reluctantly-asking-for-help-how-to-handle-huge-data-files/48959 "2021-07-27T09:54:57Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Trying\_to\_learn](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/trying_to_learn/32/28400_2.png) [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)\
**Post date:** [27 July 2021 09:54 UTC](https://discourse.nodered.org/t/reluctantly-asking-for-help-how-to-handle-huge-data-files/48959/1 "2021-07-27T09:54:57Z")

</div>

Maybe because the "log files" are Node-Red generated I should ask here.

For the past 18 months - or there about - I have been logging the temperature every.... 30 seconds.

The file is text and is time stamped.

So an extract of the file looks like:

```auto
2021-04-30 23:55:14 22.00 Deg C 22.00 Deg C
2021-04-30 23:57:22 21.00 Deg C 21.00 Deg C
2021-04-30 23:59:27 21.00 Deg C 22.00 Deg C
2021-05-01 00:01:30 21.00 Deg C 21.00 Deg C
2021-05-01 00:03:35 22.00 Deg C 22.00 Deg C
2021-05-01 00:05:40 22.00 Deg C 22.00 Deg C
2021-05-01 00:07:40 21.00 Deg C 21.00 Deg C
2021-05-01 00:09:51 21.00 Deg C 21.00 Deg C

```

(I now see I messed up with something, but that's by the by in the scheme of things)

So you have a date; time and temperature.  
And ok, they are happening every 5 seconds.

So I want to handle these files and draw graphs with "averages" depending on how much you "zoom in".

Or the other way when you "zoom out".  
So (going with the first one) you would see averages for the week - for the month. So you would see 4 weeks.  
Zoom in and it goes to days. You would see the days.  
Zoom in again and you would see the 24 hours for the day.  
Zoom in again and you see the minutes  
Zoom in again and you see the seconds.

At each level the averages are shown. "Averages"... Not sure if that is the correct word.  
There are: averages; means; and other terms I don't know.

This is not something I do, so although I have the idea of what I want to do: actually getting it happening is a whole other kettle of fish for me.

Thanks folks.

---

<div class="post-metadata">

**Author:** ![mbonani](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/mbonani/32/10631_2.png) [@mbonani](https://discourse.nodered.org/u/mbonani)\
**Post date:** [27 July 2021 10:01 UTC](https://discourse.nodered.org/t/reluctantly-asking-for-help-how-to-handle-huge-data-files/48959/2 "2021-07-27T10:01:44Z")

</div>

I think you can use InfluxDB + Grafana to achieve that

---

<div class="post-metadata">

**Author:** ![ghayne](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/ghayne/32/39_2.png) [@ghayne](https://discourse.nodered.org/u/ghayne)\
**Post date:** [27 July 2021 10:12 UTC](https://discourse.nodered.org/t/reluctantly-asking-for-help-how-to-handle-huge-data-files/48959/3 "2021-07-27T10:12:49Z")

</div>

> [@Trying\_to\_learn](#):
>
> I have been logging the temperature every.... 30 seconds.

Is that really necessary?

---

<div class="post-metadata">

**Author:** ![Trying\_to\_learn](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/trying_to_learn/32/28400_2.png) [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)\
**Post date:** [27 July 2021 10:28 UTC](https://discourse.nodered.org/t/reluctantly-asking-for-help-how-to-handle-huge-data-files/48959/4 "2021-07-27T10:28:31Z")

</div>

It is actually every 5 seconds... But who's counting?

That is because I want/have a _real time_ display of the temperature.

I just stuck the logging at the end.

I have no experience with InfluxDB and/or grafana....

Seems like I am in for some learning.

---

<div class="post-metadata">

**Author:** ![ghayne](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/ghayne/32/39_2.png) [@ghayne](https://discourse.nodered.org/u/ghayne)\
**Post date:** [27 July 2021 10:41 UTC](https://discourse.nodered.org/t/reluctantly-asking-for-help-how-to-handle-huge-data-files/48959/5 "2021-07-27T10:41:40Z")

</div>

> [@Trying\_to\_learn](#):
>
> That is because I want/have a _real time_ display of the temperature.

Actually, you can never have that due to the speed of light 😉

---

<div class="post-metadata">

**Author:** ![Trying\_to\_learn](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/trying_to_learn/32/28400_2.png) [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)\
**Post date:** [27 July 2021 10:45 UTC](https://discourse.nodered.org/t/reluctantly-asking-for-help-how-to-handle-huge-data-files/48959/6 "2021-07-27T10:45:10Z")

</div>

Way back when, I just got the sensor to spit out the reading every 5 seconds.  
It is displayed on an LED screen and ...

anyway.

I may have to learn a language to _parse_ the file and generate the needed files for the time span required.

That may be easier. It isn't I need all the time access to the data. I just have the log and want to see if there are any _trends_ that may need further research to what is happening.

---

<div class="post-metadata">

**Author:** ![ghayne](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/ghayne/32/39_2.png) [@ghayne](https://discourse.nodered.org/u/ghayne)\
**Post date:** [27 July 2021 10:49 UTC](https://discourse.nodered.org/t/reluctantly-asking-for-help-how-to-handle-huge-data-files/48959/7 "2021-07-27T10:49:11Z")

</div>

As @mbonani said, InfluxDB and Grafana are ideal for this sort of thing. It's what I, and a lot of others here, use for this sort of thing.

---

<div class="post-metadata">

**Author:** ![Trying\_to\_learn](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/trying_to_learn/32/28400_2.png) [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)\
**Post date:** [27 July 2021 10:50 UTC](https://discourse.nodered.org/t/reluctantly-asking-for-help-how-to-handle-huge-data-files/48959/8 "2021-07-27T10:50:21Z")

</div>

Thanks.

I shall have to dive in to that world and learn about it.

---

<div class="post-metadata">

**Author:** ![Paul-Reed](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/paul-reed/32/66906_2.png) [@Paul-Reed](https://discourse.nodered.org/u/Paul-Reed)\
**Post date:** [27 July 2021 13:04 UTC](https://discourse.nodered.org/t/reluctantly-asking-for-help-how-to-handle-huge-data-files/48959/9 "2021-07-27T13:04:01Z")

</div>

I'm surprised that you are only measuring the temperature to the nearest integer 😉

---

<div class="post-metadata">

**Author:** ![davidz](https://avatars.discourse-cdn.com/v4/letter/d/a88e57/32.png) [@davidz](https://discourse.nodered.org/u/davidz)\
**Post date:** [27 July 2021 14:48 UTC](https://discourse.nodered.org/t/reluctantly-asking-for-help-how-to-handle-huge-data-files/48959/10 "2021-07-27T14:48:38Z")

</div>

From the data you posted, the interval is every two minutes instead of 30 seconds?

Yes InfluxDB plus Grafana will do the job well.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [10 August 2021 14:49 UTC](https://discourse.nodered.org/t/reluctantly-asking-for-help-how-to-handle-huge-data-files/48959/11 "2021-08-10T14:49:33Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.