Told how? Linux tends to fill spare memory with things like filing system cache. You need to look at top or better htop or atop.
fs-ops --> memory node.
That was I get a real time display.
I don't know how to use top (or any version there of) to get a CLI output which I can use.
The memory node won't give you enough information. And the real-time aspect isn't as helpful as you think.
Use apt to install htop. You can sort the output by mem % and see the total memory usage.
Glances is an even more advanced tool that is really good but unfortunately it is quite memory hungry itself.
1 Like
Thanks.
Ok, they it isn't doing what I want (or thought it does).
I have the CPU load and temperature displayed. I stumbled on the fs-ops / memory node and added it to try and make a more complete picture of what the machine's state is/was.
As they are headless and mostly I am not connected to them: I am open to how to get information from the programs so I can look at it and check for trends or something like that.
This is a screen shot of one of the machines.
Andrew, I thought your pi was headless? You have vnc & lxpanel running. Lxpanel is consuming lots of memory (20%).
Try disabling vnc or disable the entire GUI from starting up in raspi-config
1 Like
Yes, but being able to VNC into them makes my life a lot easier.
Though this one isn't running BUSTER (Stretch from memory) The one before BUSTER.
Now and then I need to do things and it is a lot easier to do it via the GUI than a CLI - (Terminal).
The other two are BUSTER.
They have USB sticks plugged into them and if I don't load the GUI the USB stick/s aren't there.
If I load the GUI: they are. Over my head.
So I need the USB sticks to write logs to and to reduce SD card read/write and wearing them out.
I can't find a simpler solution.
Oh, and I also do check them via VNC now and then too, as it is easier to do things there than from the CLI.
My two cents - like Andrew - I like access to a GUI so all my Pi (Inc Zeros) run full desktop and vnc (but I don't have reboot/memory problems other than once every few months type ones)
Just for comparison - this is one of my PiZero (it just controls kitchen cheer lights so it's lightly used)
Oh dear. I can already see from your picture that you have a couple of issues.
First issue - get rid of the desktop. It is taking up masses of resources and really not giving you much. The Pi 3 isn't powerful enough for doing everything you are asking of it.
To get performance history, you will be a lot better off with InfluxDB, Telegraf and Grafana, some of which you might already be running. There are lots of examples I've previously posted of system performance data obtained this way and I even run all of that on my old Pi2 quite happily.
As for the USB storage, that should just work without any issues and without a desktop running. However, you probably need to mount the filing system. That can be automated as well. Plenty of tutorials on that I've no doubt.
The next problem is certainly related to your original question more directly though. Your node-red setup is occupying >35% of memory but it is only 153M of resident memory and 252M of virtual.
On my Pi3, influxd is my 2nd largest single app using 136M or resident memory and 1048M of virtual - but it is only using 14.7% of the memory. My largest is the Ubiquiti Wi-Fi controller which runs under the monsterous Java and uses 253M of resident memory, 1297M of virutual and even that is only using 25% of memory.
This tells me that I think you are using an original Pi3 A+ model with just 512MB of RAM. Please run grep MemTotal /proc/meminfo for us so that we can confirm.
I believe that you have also disabled SWAP. While this saves some wear and tear on your SD-Card, it really isn't a very good option if you are running low on RAM and I believe this is what is causing the Pi to crash.
I've got 100MB of SWAP on my 1GB Pi3 and it gets used - mainly because of the Ubiquiti Java-based controller app. Bear in mind that this was my main server until recently and it runs the controller (which also requires MongoDB), a big InfluxDB database, node-red, telegraf, mosquitto, grafana, some bluetooth stuff. All without any crashes - for years!
Which has SWAP turned on 
Incidentally, you don't have to have a desktop running all the time in order to make use of VNC. I don't.
OK, It didn't quite tell the truth there - I'd forgotton that I'd broken my golden rule of not using Docker! In fact, I run rdesktop in a Docker environment.
But I only run it on my "new" laptop-based server which runs headless Debian Buster. It has 8GB RAM, a 256GB HDD and an Intel i5 processor (4-cores). Even then it isn't exactly speedy.
In reply:
P.S. (after thought / editing)
This is the .... main machine. The htop above is for another one.
I'll stick to the main one for a moment though.
pi@TimePi:~ $ grep MemTotal /proc/meminfo
MemTotal: 443080 kB
pi@TimePi:~ $
The machine is a RasPi 2 Rev B from what I can get from asking it.
It has the yellow composite video output and less than the usual number of GPIO pins.
And only 2 USB ports.
On the logging stuff / history.
I've tried using databases and just run into problems. I seemed destined to never being able to get my head around them.
What I have is a couple of commands to ask for CPU load, temperature - and now free memory.
They are displayed on the dashboard's gauge and chart and the memory usage is with ui_level.
That stuff is (well, still isn't as I haven't actually turned it on) written to a text file so I can - if I want - go back and check things. Will I ever? Probably not. But it wasn't much to add and isn't mission critical for memory usage.
Now onto the RasPi Zero (W):
pi@TelePi:~ $ grep MemTotal /proc/meminfo
MemTotal: 378116 kB
pi@TelePi:~ $
So it is in a worse state than the first one for memory.
This machine does a lot less.
All it is for is to kind of monitor the main one and if things go wrong: give me an indication.
It has a USB stick plugged into it and it DOES log stuff. But that is a whole other set of data.
The LED strip isn't a NEOPIXEL, but POMORONI one. Same but different.
Addressable RGB.
If the main machine detects an error/problem it sends a message and the Zero turns on an LED.
That is about all it does.
The third one (as yet not mentioned) is a RasPi 3B and has a USB stick. All/any logging is done to it.
It is my DNS server and controls a few external hardware devices. It does a bit of work.
OK, so even worse since this is a Pi2 with just 512M RAM.
You need to lighten the load on that device and try turning on SWAP of at least 100MB.
Is there a way I can set the swap to use USB memory rather than SD?
There is a spare USB port and I have a 16Gig USB stick doing nothing just now.
That is the Zero you are talking about - yes?
Ok, can I use the existing USB stick's memory as swap file?
Sorry... As I said, the screen shot is from the ZERO (W).
Which one has the SWAP turned off - the one that is crashing?
Yes - but I don't recommend it. I said before about using a decent SD-Card with plenty of room. That will have wear levelling which will save you from terminal card corruption. A USB still almost certainly will not have wear levelling and will likely be slower even than the SD-Card (especially on the USB bus of the Pi which is shared across all USB ports and the network I believe).
Ok sorry.
(One machine at a time)
AFAIK I am using name brand good quality SD cards.
(There is a whole other story there, but I won't go there now. It will only confuse the situation.)
I actually thought I had turned the SWAP off on all of them.
Though the main one did die a fair while back and I maybe didn't disable it on there.
But I haven't shown you any screen shots of that.
(In this thread - I hope)
This is the Zero's sd card stats that I can get/give:
pi@TelePi:~ $ df -Bm
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/root 29858M 4680M 23905M 17% /
devtmpfs 181M 0M 181M 0% /dev
tmpfs 185M 0M 185M 0% /dev/shm
tmpfs 185M 21M 164M 12% /run
tmpfs 5M 1M 5M 1% /run/lock
tmpfs 185M 0M 185M 0% /sys/fs/cgroup
/dev/mmcblk0p1 42M 23M 19M 55% /boot
tmpfs 37M 0M 37M 0% /run/user/1000
/dev/sda1 1007M 234M 774M 24% /media/pi/9020-9C27
pi@TelePi:~ $
55% used - yes? (Boot)
I'll go and read up on how to activate the SWAP system.
I did document it, so it may not take too long.
That's just /boot, mine is 53% on the Buster based Pi3. The Pi2 is 39% but that is a much older version of Rasbian. On my Pi2, / is 72% full, the Pi3 is 46%. Both are using Samsung EVO 32GB cards.
Swap doesn't show up on df
So, boot is only the boot.. Is it root which is where the O/S is.
Ok, sorry.
So 17%. Ok. I am sitting good in that respect.
I have activated the SWAP on the RPZ(w) and am now doing the Raspi2B
(Done.)
And did the third one too.
It had a 100 set as the swap file and its htop was maxed out on memory use.
The website I went to for setting swap file size for RasPi had 1024.
Ok, update:
The Zero didn't seem to want to activate the swapfile.
htop showed it as not being used.
When I tried to do it again I was told it was busy.
I'm rebooting it and shall report what happens.
Machine rebooted.
swap file not seeming to be used.
This is the config file:
pi@TelePi:~/Documents $ cat /etc/dphys-swapfile
# /etc/dphys-swapfile - user settings for dphys-swapfile package
# author Neil Franklin, last modification 2010.05.05
# copyright ETH Zuerich Physics Departement
# use under either modified/non-advertising BSD or GPL license
# this file is sourced with . so full normal sh syntax applies
# the default settings are added as commented out CONF_*=* lines
# where we want the swapfile to be, this is the default
#CONF_SWAPFILE=/var/swap
# set size to absolute value, leaving empty (default) then uses computed value
# you most likely don't want this, unless you have an special disk situation
CONF_SWAPSIZE=1024
# set size to computed value, this times RAM size, dynamically adapts,
# guarantees that there is enough swap without wasting disk space on excess
#CONF_SWAPFACTOR=2
# restrict size (computed and absolute!) to maximally this limit
# can be set to empty for no limit, but beware of filled partitions!
# this is/was a (outdated?) 32bit kernel limit (in MBytes), do not overrun it
# but is also sensible on 64bit to prevent filling /var or even / partition
#CONF_MAXSWAP=2048
And I entered the command
sudo dphys-swapfile swapon
before the reboot.
And this is the main machine - now running with swap file.
Well, though not conclusive: It has been a few days and the machines have been a lot more well behaved then they were before.
Things are looking good.
--
What I did: activated swap files on the RasPis and made the size 1024, as I had disabled it and on others it was only 100.
Not sure if it is the 100 -> 1024 which was the solution or just activating it on the ones which didn't have the swap file active.
I know it requires patience, but it's generally better to make changes one at a time.
Inevitably that would delay any crash/reboot anyway because you've now got more memory to play with. Quite possibly it also lets Linux manage memory better as well.
Long-term testing will be needed.
1 Like
