Hi again, In past I had quite a lot of problems (which I thought I solved) with consuming too much memory (running out of heap memory). After understanding that all the graphs that I was generating ended up eating all available memory. Since then I have started using Influxdb (V2/Flux) and transferred all graphs to Grafana. Then pulling them up in iFrame in "external" flexdash screens. The memory is working "well" as I do not have any errors on NodeRed nor Have I any issues when running the screen from PC browsers.
But Android tablets seems to be effected with this incremental memory increase which occurs as node red session gets "older". On systemctl restart my system starts now about 260 mb of memory and after gradually 2-3 days its around 300-350 mb. On the attached picture (grafana) you can see memory increase. The issue is that at somewhere 330-340 mb limit the android (10/11/12 all effected) start to do constant refresh of the screen and also at that time any button press / execution from the tablets come very sluggish.
The curious part is that it seems that the heavy memory up/down ( I guess V8 doing garbage cleaning) starts almost 14:00 every day and stops at 03:00 next night. System is not doing anything extra at those times, all data is gathered 24/7 in same way. And my usage do not start typically before 17:00 and clearly stops before midnight. System logs in Ubuntu 20 do not reveal anything special happening on those times.
I originally thought that my old 16Gb / 1GB RAM tablets were the bottle neck, but upgrading them to either 2GB or 4GB RAM memory had no effect. All behaved similarly. Also type of browser had not/or very little effect (Chrome, Edge, Samsung Internet, Firefox).
I have also run the "inspector" and the tried to identify the issues in Chrome / memory handling, but can not really seems to find any obvious culprit in the system.
Has anyone seen similar effect and any other ideas to resolve the issue ( now contemplating putting a cron job to restart the node red every 2 nights... a way forward but not really a solution
security@SAMONE:~$ node-red version
12 Aug 21:23:53 - [info]
Welcome to Node-RED
===================
12 Aug 21:23:53 - [info] Node-RED version: v3.0.2
12 Aug 21:23:53 - [info] Node.js version: v16.20.1
12 Aug 21:23:53 - [info] Linux 5.4.0-155-generic x64 LE
12 Aug 21:23:53 - [info] Loading palette nodes
12 Aug 21:23:54 - [info] Node-RED FlexDash plugin version 0.4.154
12 Aug 21:23:54 - [info] Node-RED Vue version 0.1.15
12 Aug 21:23:54 - [info] Creating Vue template for 'flexdash dashboard' from /home/security/.node-red/node_modules/@flexdash/node-red-flexdash/flexdash-dashboard.vue
12 Aug 21:23:54 - [info] Node-RED FlexDash version 0.4.154
12 Aug 21:23:54 - [info] Node-RED FD Core Widgets version 0.4.64
12 Aug 21:23:54 - [info] Creating Vue template for 'flexdash container' from /home/security/.node-red/node_modules/@flexdash/node-red-flexdash/flexdash-container.vue
12 Aug 21:23:54 - [info] Creating Vue template for 'flexdash tab' from /home/security/.node-red/node_modules/@flexdash/node-red-flexdash/flexdash-tab.vue
12 Aug 21:23:54 - [info] Creating Vue template for 'flexdash custom' from /home/security/.node-red/node_modules/@flexdash/node-red-flexdash/flexdash-custom.vue
12 Aug 21:23:55 - [info] Dashboard version 3.5.0 started at /ui
12 Aug 21:23:55 - [warn] ------------------------------------------------------
12 Aug 21:23:55 - [warn] [node-red-node-rbe/rbe] 'rbe' already registered by module node-red
12 Aug 21:23:55 - [warn] ------------------------------------------------------
12 Aug 21:23:55 - [info] Settings file : /home/security/.node-red/settings.js
12 Aug 21:23:55 - [info] HTTP Static : /home/security/Pictures > /
12 Aug 21:23:55 - [info] Context store : 'memoryOnly' [module=memory]
12 Aug 21:23:55 - [info] Context store : 'file' [module=localfilesystem]
12 Aug 21:23:55 - [info] Vue 'fd-container-config' from /home/security/.node-red/node_modules/@flexdash/node-red-flexdash/components/fd-container-config.vue
12 Aug 21:23:55 - [info] Vue 'fd-fmt-node-label' from /home/security/.node-red/node_modules/@flexdash/node-red-flexdash/components/fd-fmt-node-label.vue
12 Aug 21:23:55 - [info] Vue 'fd-general-tab' from /home/security/.node-red/node_modules/@flexdash/node-red-flexdash/components/fd-general-tab.vue
12 Aug 21:23:55 - [info] Vue 'fd-grid-sorter-item' from /home/security/.node-red/node_modules/@flexdash/node-red-flexdash/components/fd-grid-sorter-item.vue
12 Aug 21:23:55 - [info] Vue 'fd-grid-sorter' from /home/security/.node-red/node_modules/@flexdash/node-red-flexdash/components/fd-grid-sorter.vue
12 Aug 21:23:55 - [info] Vue 'fd-list-sorter' from /home/security/.node-red/node_modules/@flexdash/node-red-flexdash/components/fd-list-sorter.vue
12 Aug 21:23:55 - [info] FlexDash UI version 0.4.90
Not quite clear - is Node-RED running on the tablet as well as the browser? If not, where is Node-RED running?
Also, I assume you are using other features of FlexDash, not just embedding Graphana in it? If not, that is a lot of overheads for no gain. Indeed, since Grafana is web-based, would it not be easier to simply have a (menu-type) link to Grafana directly and eliminate the embedding? You could add a link to the Graphana dashboard too that would go back to your menu page to handle users moving between multiple Grafana dashboards.
Apologies not been clear enough. The Node Red is running on a dedicated Ubuntu 20.04 LTS.
On the server side I do not have any issues. It is really more the android tablet side I do see the issues now. My understanding was as the current dashboard (3.5.0) is a SPA I needed to off load some of the burden to other "outside" dashboards so that why I put in play the Flexdash (0.4.154) . My understanding was that as Flexdash sits in a different address these frames would not be loaded as part of the SPA to the tablets when initiating the normal dashboard (3.5.0).
So hence I have created following Flexdash pages .
Using Iframe is more convenient as I can have different Grafana time windows displaying selected data in the same screen (and scales nicely in tablet or phone).
Has my understanding been wrong and this is burdening node red (and also hence the SPA/ tablet) memory usage just us much still?
Yes, I think it is, I'm afraid. As far as I can see, Flexdash is merely an overhead here and not doing anything useful. AFIK, Flexdash is also an SPA?
In any case, you can easily set up multiple Grafana dashboards with different time windows. As everything is a web app, switching between Grafana Dashboards is still only an HTML link away. That could be done with a simple http-in/-out paid with a simple menu.
I can't say that switching to use Grafana more directly would solve your issue but I can't see that Flexdash is helping you. I can't even recommend using uibuilder here!
Thanks for insight. I have now removed Flexdash and incorporated direct links to grafana (though still in iFrame link in tab), hence moving a little bit of overhead from my node red setup. Lets see if that helped in any significant way.
Eagerly waiting for the new dashboard version to grow from alpha version.
I'm confused by your description of the problem. You show graphs about memory usage in NodeRED, which is on the server and at the same time you write about memory usage on android tablets. Those are completely unrelated things...
If you're just using FlexDash to embed Grafana then that's indeed not worth it and just added complexity. Why don't you just use Gafana as-is then? It has a nice drop-down selector to switch between its "dashboards".
FYI: Influx with Flux is an absolute performance and memory shit-show. I tried to use it and was completely appalled at the performance, it's worse than I could have ever imagined. (I have used time series DB for decades and have written a highly scalable on myself, so I do have some background.)
Hi tve, Thanks for your comments let me try to unconfuse you ( long post but please read it through).
So my journey with node-red start really from version one long time ago when I built my first nodemcu arduino boards to measure temperature originally inside my house. Node Red was new and ideal on gathering and showing the data in one place.
As skills got better (in arduino side also) along came Humidity and CO2 sensorin, linking that with HVAC and inside and outside temperatures.
Next I started to add also weather forecasting on top of the system.
Controlling lights (Ikea and Tyua) and other electronics like Radio & TV were next on the list.
As that was all in place and working nicely the energy consumptions came next. Adding Shelly and other measurement devices came along.
And this is where my problems originally started. Gathering a lot of data and plotting that through Node Red was consuming too much memory.
The Ubuntu server I'm running has enough squeeze (AMD Ryzen 5 / 16 GM memory and 2TB SSD) so the physical components were never the problem on server side. But as I had been using the original Node-Red ver.1 Ubuntu installation script it had left a minor line in code that only allowed me to use 256 MB on that whopper 16 GB memory. This problem my friends here helped me to identify and correct on my path of learning. That solved part of the issue.
During this time I also moved using influxDB and Grafana to do the heavy data gathering and plotting.
I though still saw periodically that my two separte android tablets ( that are typically showing these screens; out of total 18 pages/tabs) where reloading (from scratch) the screens. As one is in living room next to television set it started to bug me that system was not really working as it should.
As the memory consumption (or the setup of it) was a problem in server side I was thinking might it be also the culprit for these constant reloads. Hence I used one of the node red options to track how much I'm actually consuming memory on server side and what (if any) correlation to the refresh on android tablet side. And behold there was. As pointed when server side was hitting the 340-350 MB levels the tablets started to do these reloads. Restarting the Node-Red cured the problem for couple of days but it always came back (through memory leak /creep). So reducing memory usage in Server side came as one possible options. That time again my good friends here in forum teached me here the nature on current dashboard version (3.5.0) that it was indeed a SPA model meaning that every TAB ( or page ) that I had at that time would be actually loaded into the tablets memory (and not only the one which was shown there). I Tried the FlexDash as I believed (again my friends corrected me) that that would take away memory allocation issue and eased up the burden on the android tablet side. Ended up removing FlexDash and now using directly the Grafana dashboards via Iframe setup.
You can see from the memory chart that actually removing the FlexDash has made the system use drastically less memory (around 220 mb) now for 3 days and hence the tablets have also behaved "correctly"
So even though the issues are on server side they are reflected to the tablets and that was the point I wanted to raise (or question that had anyone else noted this). Upgrading the tablet to 4GB memory had no effect , nor trying other web browsers.
Last comment about InfluxDB vs. Flux. Indeed I was surprised that the Grafana point and click did not work any more. But as I had converted my databases to new 2.0 version already I was "committed" , hence learning a new coding ritual that was needed to Grafana to understand the data correctly. Main advantage that was my reason to go to Flux is the correct handling of a Month (1mo) , not 4 weeks or 28/30/31 days.
Sorry for the long post and hope it helps to understand my continuous journey of learning in Node Red world.
Happens typically same time for both screen (even though presenting different pages). I guess this is due SPA model.
Like for the 1st screen the pages flashes totally black (or background grey) and the values are missing for a second. Then they reload to last values. The bottom right which is wind direction and speed meter (SVG animation) requires a update every 15 second so that part is waiting for a new value and then updates it.
Similarly the 2nd screen is flashing blanc and then reloading the last values. The bottom right is Grafana Iframe embedding and it takes couple of seconds to reload the graphs.
As my memory usage is still under 240 mb at the moment I can not reproduced it to record a video. Hopefully you get the understanding from description.
Have a look in /var/log/syslog for the time at which it happens and see if there is anything relevant there.
[Edit] Do you mean it just happens suddenly without warning, and without you doing anything? How is it that you manage to notice it if it only happens rarely and is over in 15 seconds?
[Further Edit] Looking back I see you have already answered those questions.
What does top show when the memory is high?
On the chart at the top you showed a green line which purported to be CPU, but it is absolutely constant, which is odd. What does top say about the cpu used by node-red?
As stated in original post, unfortunately the system is not showing anything untypical at those points. Same activities (logged ones) happen around the whole day and nothing steps up from those moments when the memory peaks and issues occur.
Blockquote [Further Edit] Looking back I see you have already answered those questions.
What does top show when the memory is high?
On the chart at the top you showed a green line which purported to be CPU, but it is absolutely constant, which is odd. What does top say about the cpu used by node-red?
The green and red solid lines are my own illustration showing the good / bad levels. CPU is not on those. When I was measuring it it actually never hist even 2-3% levels on CPU usage.
Hmm. I was just running Htop or system Monitor the from CPU usage and zoneminder (security camera system) is typically highest users, Node red tends to be very low.
From sorting via memory usage Node-Red is always in top 5, like just now (but as you see memory usage now much lower consistently and tablets are "behaving" correctly.
