Thanks @HaroldPetersInskipp!
My RPi is a 2 GB version, which I picked based on the assumption - I run it head-less, why would I need more memory (you spot the layman, right?)!?
I very much appreciate you sharing your stats! It is quite possible that my bottle neck is actually memory (and my pi is busy with swapping rather then executing code).
The air quality monitor is just a test. my average mqtt load is 0 to 2 msg/sec and I only have occasional peaks of ~35msg / sec. I have NOT narrowed it down to what might cause the issue as I didn't see this causing the main load (but maybe it is ...).
htop shows consistently nod red as the "top consumer":
I have grafana and influx db also running in the background but they don't do anything yet. My homematic installation runs on another pi but it brings in "all values" as default of which I am maybe only using 1% ... not sure if that could be somehow limited but my aim would be in the long rung to completely move away from homematic and switch all devices over to something else ... but that will take time.
I am also running a usb / serial connection to my home built arduino based alarm system, but again, on average only 1 msg/min (but peaks, when something happens).
I also run a couple of Web based services to 2 weather (forecast) services, operate my wallbox ev charger via their web UI (one call every 30 seconds) and have our BMW i3 connected for charging control and pre-conditioning (via python script, which when it triggers runs for a longish time but I don't think the processing load is high though) and showing the location of the car on a map. I am also fiddling around with our hybrid Merc via WebAPI but that is not yet up and running.
My pi zero based front door web came is dumping an image each second on the PI's SSD which is then further processed by logic in my home automation system (when an event triggers, I can se the previous 20 seconds of activity and send potential telegram message, email, and long term archiving of "motion events").
I also have 4 web api based shelly dimmers connected via shelly nodes. I have no idea about their "load".
As I don't have homeassist I do all lights and ventilation "automation" with manual "logic". I am heavily using norelite for that and as cool as it is, I have my suspicion that it is also cpu intensive ...
I also communicate to 3 Google home devices and check the availability of some (core) appliances frequently (door bell down, nas down, front came not available, pi-hole going down etc, etc.).
Long term trending is done via sysvar to the homematic based historian. I'm sending some 250 event driven values for long term archiving (frequency is hard to say - some go every second, some only once an hour or maybe even less but hard to say!).
And as I had said, all "worthy" notifications are sent via telegram (and email) from where I can also control a considerable amount of devices via a self define command line logic (my VPN for remote access has only been up and running more recently).
Regarding memory I am not a linux guru - If I interpret HTOP correctly, I think it says it uses only about 50% of my 2 GB available, right?
"free" shows this:
When I switched form my RPi 3B to my 4B some time last year, it was a big performance boost. But with all the stuff I have been adding, I am kind of back at where I was then ... it simply annoys me when I have to wait 15 seconds before a changed flow goes life.
I dare to say my CPU average load is similar to yours with double the amount of nodes (but no homeassist or worth while grafana / db services yet running in the background).
If I would spend enough time in optimising my over the last 2 years grown installation, I am sure I could squeeze out a bit more performance, but can't see this solving my issue if I once more would like to "double the load" (my heating ebus communication and pi-hole run on a separate device, I am not yet done with further air quality monitors and the associate automation, would like to do more heating profile optimisation [currently runs mostly on the homematic thermostats but would like a more centralised approach], am planning the integration of our ring door bell, also wanted to change the communication with our ev wall charger from web api via provider website to a local ocpp bases server client architecture, etc. etc. ). And, as I did admit, I am not a javascript / linux trained programmer (actually, my history is with Simatic logic controllers, a very different beast 
)
If someone has a script, that can easily (temporarily) deactivate ALL ui_charts in runtime, I would be very interested to try and see if my suspicion is correct. Or if someone can confirm that memory is my culprit, I might try an 8GB Pi 4 (if they become available again ...).
But otherwise I think that my easiness way forward will be a more powerful hw platform (or yet more distributed services across other PIs which I don't want).
THANKS
