Splitting a konstant data stream into 200 sub streams

hello
i am turning to you again today as i need an idea on how to implement a flow reasonably, briefly and efficiently.
I get a constant stream of values via a single function.
This is several hundred a minute, but they repeat in different orders.
I can tell them apart because there is a number in the payload of which the last 3 digits (001-255) can be assigned to a data source.
I get a true or false of each value if it is active and some more process data in form of different named values.
Now I want to split this data stream to about 200 LED, text and gauge. Of the 255 possible data sources not all 255 are used today, there are free areas in the process chain.
But one function with 200 outputs is as horrible as 200 single functions.
It is important that I can take over the payload of the current data stream (the number) in the text field so that I can see which process is standing or see on the gauge whether it is slow.
I already thought about making it block wise 0-99, 100-199, 200-255 but even then I have a huge number of functions.
I look forward to your thoughts and ideas on how I can electively convert this problem in node red. It remains 200 times 3 outputs for the dashboard

Can I ask where the data are coming from? There may be something that can be done upstream.

Otherwise I would send the values on to MQTT using topics that include the source id. That can be done in one function node with one output, and passing it to an MQTT node.

Then the display widgets can be fed from MQTT In nodes that provide just the data they require. I use MQTT between all my process data and logic, and dashboards. Doing that means that the dashboard is isolated from everything else, and can even be run in a different machine, provided it can access MQTT. In fact I use it between my IO and system logic too, so the system is made up largely of:

  • IO flows that receive data and send it to MQTT, or get data from MQTT and send it to the outputs.
  • Dashboard flows that display data from MQTT and pass operator input back into MQTT
  • Logic flows that interact with the IO and Display MQTT topics to perform any system logic,

Getting back to your immediate problem, if you don't want to use MQTT then store all the data in a global context structure, possibly using an array with the source id as the index. Then separately fetch the data from that structure for display, much like using MQTT, though it is a more complex way of doing it.

1 Like

Hello colin.
That's what I love about the forum, you get a completely new idea and perspective.
I steal the data from a native bus system which works similar to the knx system, of course not compatible. Someone here once built a wild C++ script that read the data and wrote it to a database, but it was discontinued because it filled up within days and no one worked with it. Unfortunately I can't get the database to work because I don't know the hardcoded user & PW :frowning:
I now just want to put a pi, node red with dashboard for faster reaction to errors.
LED Red: Error in process chains NR xxxxxxxx
LED Green, everything is running .
Expandable without end, but must first prove this as a poc and see if it is used at all or lies in the corner after a few days.

EDSTOBI