No criticism intended here, just interested observations:
My first thought looking at that flow diagram was that you have an awful lot of function nodes.
On reflection, each of your functions is probably relatively simple, so no doubt more easily maintained than a single complex function.
I do wonder why you have no switch nodes to route messages along one wire or another based on the message properties, nor change nodes to adjust the message structure for your requirements.
It looks like your output may be multiple messages, each with a single item of data.
I wonder why you need this?
For example, a dashboard gauge can as easily display the value of msg.payload.something[0].temperature (or from the same message, msg.payload.something[0].dewpoint) as it can msg.temperature.
It would be interesting to see the whole flow exported, if you are happy to share it.
msg is just a javascript object. If the code modifies it then it is modified.
To concur with @jbudd's comments, it appears you are bringing all the data in from different devices under the same topic and then examining the data to work out what is in it. That is horribly inefficient. For example you will be checking all the properties of messages to see if they are ATC*, when you know that they are actually from a different device type. Either have a Switch node after the MQTT node, testing the topic, and splitting the message down different paths dependent on device type, or have multiple MQTT In nodes, one for each device type. I don't think there is a significant difference in efficiency between those two approaches.
I do have other flows where I use the "simpler" switch, change etc. nodes. But I have found that albeit being super simple in the beginning, they quickly become bottle necks.
So I decided to try to do as much as possible with functions. This has the added benefit of being able to learn much more and perform more complex tasks in the future
The debug nodes are actually not my real output. They are really just for debugging right now. The output will later be an mqtt out node.
So in the current stage of my little project, I am separating the data for processing.
This will be followed by a block of functions that will process the data.
Once processed, the different streams will be re-joined to form one output message (per device... see more on that in my response to Colin)
Perfectly valid points
Maybe I need to elaborate a bit more.
Device types are different hardware types (usually linked to different manufacturers).
So I have:
Shelly 1PM, Shelly 2.5 and Shelly 3EM.
Gosund SP1 and Gosund SP112.
Refoss P11
Xiaomi Mijia Thermometer & Hygrometer
eq3 Thermostat (heating valve)
Devices are then "one piece of hardware" no matter who the manufacturer. Entities are then e.g. power sensor, energy total sensor, power switch etc.
The first three device types are running with tasmota firmware. So they will all have the respective device_id in the topic, e.g. "tele/tasmota_12345678" which is based on the MAC of the devices.
These run to the bottom flow because they are all power sensors and their messages are all very similar.
So the first function checks if it is one of these device types. Since the topic contains the device id, I do not need to split into devices anymore. The second function then splits into the entities Energy Power, Energy Total and Energy Today.
However, those with multiple outputs (e.g. Shelly 2.5 or Shelly 3EM) will need separate processing of these 3 outputs, hence then another function splitting into the entities Power[0] to Power[3].
The Xiaomi and eq3 are in fact bluetooth devices with their own firmware (whose behavior I cannot really influence). They are relayed through an ESP32 dev kit board running tasmota.
So the data comes in via bluetooth to the ESP32 and the ESP32 then relays/publishes the data as MQTT messages via Wifi.
Each Xiaomi device uses "ATC" + a unique ID based on MAC to identify itself to tasmota.
Tasmota does not use the same topic as it does for the Shelly/Gosund device types. This alternative topic is not device specific but device type specific. Meaning, all Xiaomi Mijia devices' data are published via one large message with just one topic (they may be split into multiple messages if there are more than 4 Xiaomi devices).
So I need the top flow to check the message for "ATC" and not the topic.
The same applies to the eq3 valves, which I have not implemented yet because they are not used in summer (and are essentially clone nodes of the ATC flow).
Maybe as an additional note:
I could of course join different functions into one and have less processing steps. But right now this makes it easier to work out mistakes and to follow the processing. IF this leads to performance problems later, I will merge them into fewer but larger functions.
Until then I think it is easier for you to support and me to learn if I keep each step in a separate node.
P.S.:
Of course I am happy to share the final glow later. It is currently in a private github repo but once I have a more refined and more feature rich flow, I will happily share it with whoever is interested.
I expect you are already aware that a function node can have multiple outputs. That can avoid sending messages down paths that are not relevant to the message type.
I could have joined the two and used two outputs, but I am not sure if there is any benefit of doing that.
Downside is a larger first joint function to separate device types.
This I have clean paths and the only thing they have in common is the input mqtt node.
Has the additional benefit that I can give them separate mqtt in nodes if I need to subscripe to a different topic.
Right now it is one node for one task for one path.
But it only truly becomes efficient if I do something like for each [i] in msg.payload.Energy.Power[i].
Syntax probably wrong, but I was trying to code-explain
Plus, this would probably not handle the case where Power does not have sub-properties but is only msg.payload.Energy.Power (without [i]).
But I then still have to define the number of outputs which makes the whole thing futile again. Only saves nodes then
That is not entirely true. Every time you have more than one wire leaving a node, clones of the msg are generated. That is possibly of little consequence here but it can make a difference if high volume is also a factor.
As I said, one device type would need to be switched based on topic. The other two need to be switched based on elements in the message.
So yes, I could add a switch node but I would still need the function node for the bluetooth devices afterwards (as different BLE device types will come in on the same topic). So effectively only saving one function node.
Ah, good to know. I did not consider that. But that would also be the case for switch nodes. So those would not save anything.
But it would save on wires and individual nodes if I combine as many functions as possible.
I will consider this at the end, when I know precisely which steps I need to perform
By the way, performance might be an issue.
The sensors themselves update more than once per second but I will not receive those live.
The input data per device (~50 devices right now) will be published every 10 seconds.
so my flow could receive anything from 5 messages per second (best case) to 50 messages "simultaneously" (worst case).
All messages must be processed before the next messages are received (so in less than 10 seconds).
I count at least 3 less clones and a lot less orange
And you lose the concept of visual flow programming.
TBH, I have not followed this thread entirely but based mostly on this screenshot:
... your approach is not really optimal for a low/no-code visual flow based env. I have seen some of the regulars post suggestions in this thread and they have been doing this for many years - I would hope you explore the suggestions?
FWIW, using a function node to simply route or assign something to msg.payload/topic is very much overkill. The function node runs in a sandbox (VM) and has overhead not present in the likes of a switch or change node.
EDIT
NOTE: I should point out, my suggestion above (in the scribbled screenshot) may be complete nonsense (I dont have your original flow or data to fully understand what goes on in the function nodes and what you expect at the end)
Perhaps @Steve-Mcl could have said
"If a message travels down multiple wires from a node, the message is cloned.
But with a switch node (or if so coded, a function node with multiple outputs), the message travels down one or other wire, not both."
Anyway, you are using Node-red in a way that you are comfortable with, so all is fine
Whilst I like that Node-RED makes it easy to understand what is happening because of the visual component, I never intended to use it as a solely visual tool. That's just an added bonus for me
And I do like to learn new stuff, so javascript function nodes is more fun for me
Definitely will. But as I mentioned, some nodes will remain until I have built everything because it is easier to debug 10 nodes with small functions than 1 node with a large function.
Really good to know. I was not aware that function nodes were treated differently! Is this also true if you use javascript expressions in a switch node? Or only the case for actual function nodes?
Separate post because it is a programming question:
I want to use variables in my flow. From the docs I would think flow variables as I do not need them in other flows.
Does the variable have to be part of any path or can it just be a standalone node where I define my variables?
Is there any other node than a function node even intended for this?
Change node can set them. other nodes can use them.
Not certain what you mean.
If you define a flow context variable, it is available to every node on that page/tab
If you define a global context variable, it is available to every node on every page/tab
NOTE: Be careful with context. It is tempting to use it everywhere but in truth, it can bite when you have quick successive messages (remember, node-red is async). Where possible, try to put the data to process into the msg, pass it along to the next node for further processing etc. This is akin to PURE programming. A useful piece to export is the link-call node. By its very nature, it helps you minimise code duplication and promotes PURE programming.
There are no "javascript expressions" rather they are JSONata expressions which is a kinda interpreter that runs in JavaScript. It is pretty good at what it does, but it can be slow (it runs atop JS). I personally dont use it for anything more than string concatenation / simple maths.
TLDR; JSONata is almost always slower than a function node and the function node is almost always slower than a "NON JSONata expression" change node operation like copy or move.
But function can also set them, can it not? (see my example above)
I mean, are the variables first initialized when a msg is wired into the node? Or can I just place a single change/function node anywhere in my flow with no input or output connection and it will set the flow variables?
This confuses me a bit.
I thought there were three types: flow, global and context. You call them flow context and global context. I though context was a third type which is only available with the path it is initialized in.
No*, a node has to have been executed (a msg passed in). Of course this depends on which node and what msg you send, but typically (99% time), if a node does something with context, that occurs when a msg arrives.
*Some specialist nodes might set context at invocation as opposed to "on input" - you likely wont have installed any nodes that do that.
I count at least 3 less clones and a lot less orange 
