# Complexity of join, simplify flow?

**URL:** <https://discourse.nodered.org/t/complexity-of-join-simplify-flow/94992>\
**Category:** General\
**Created:** [29 January 2025 21:22 UTC](https://discourse.nodered.org/t/complexity-of-join-simplify-flow/94992 "2025-01-29T21:22:52Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![ThingsTinkerer](https://avatars.discourse-cdn.com/v4/letter/t/91b2a8/32.png) [@ThingsTinkerer](https://discourse.nodered.org/u/ThingsTinkerer)\
**Post date:** [29 January 2025 21:22 UTC](https://discourse.nodered.org/t/complexity-of-join-simplify-flow/94992/1 "2025-01-29T21:22:53Z")

</div>

This is a slightly different topic from me. I got something that works. But it feels a bit complex, so looking for possible simplifications. Been experimenting with joins lately to combine multiple msg to a single object and send 1 http message (instead of several small ones).

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/2/b/2b571728d6ee8045f06e2f9aade9783481d67fcd.jpeg)  
This example is quite typical for collecting data and making statistics. It largely consists of 3 parts, regardless of what the data is (protocol, source), what you want to do with it (processing) or where to send it (http, file, db):

1. read data (I/O input)
2. process data
3. write data (I/O output)

First of all, when dealing with joins, I have to make sure not to lose any messages. Luckily I already have a robust custom subflow to handle modbus requests, making sure it always produce output (even if modbux flex getter originally can swallow messages silently). Then, in case something goes wrong, I send a reset message to join before processing the next batch. Because join must have all messages, I can't discard failed messages mid-stream. That's what the 2 lines at the bottom are for (keep failed messages and reset signal).

Then there is a `prepare join` function node to put everything I need from individual messages into the payload. Can be as simple as cloning the entire message into it's own payload. Lastly, there is the `prepare write` which cleans up the join and format the message according to the output requirements.

I know some may think parts/join is simple. I'm not sure I agree. By all means, it may be easy when you have all data in a small example without I/O. The difference between join and no join in this case is example is 2 additional lines (failed messages and rset) going around the main portion of the flow. A join node (which is ok, no escaping that), a `prepare join` node and some additional complexities in `prepare read`. Perhaps I'm overthinking and this is as simple as it gets?

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [29 January 2025 21:58 UTC](https://discourse.nodered.org/t/complexity-of-join-simplify-flow/94992/2 "2025-01-29T21:58:54Z")

</div>

What are the blue nodes?

[Edit] Also, are you doing lots of individual modbus queries, one for each value, and what database are you using?

---

<div class="post-metadata">

**Author:** ![ThingsTinkerer](https://avatars.discourse-cdn.com/v4/letter/t/91b2a8/32.png) [@ThingsTinkerer](https://discourse.nodered.org/u/ThingsTinkerer)\
**Post date:** [29 January 2025 22:18 UTC](https://discourse.nodered.org/t/complexity-of-join-simplify-flow/94992/3 "2025-01-29T22:18:58Z")

</div>

The blue ones are buffer parsers. Yes, lots of different modbus queries for different ranges of values. Database is in Thingsboard cloud, think it's a mix of cassandra & postgres. But we are not affected by that in any way, as we simply use the http rest api to send data.

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [30 January 2025 09:03 UTC](https://discourse.nodered.org/t/complexity-of-join-simplify-flow/94992/4 "2025-01-30T09:03:56Z")

</div>

> [@ThingsTinkerer](#):
>
> lots of different modbus queries for different ranges of values

If is generally best not to try to join data from multiple reads when saving in a database. Use different measurements (or tables, or keys, depending on the database) for each set of data fetched. In this case that means that each successful modbus read will result in a record in the database. That will dramatically simplify your flow, and you will not end up with a database with holes in it for missing data.

---

<div class="post-metadata">

**Author:** ![ThingsTinkerer](https://avatars.discourse-cdn.com/v4/letter/t/91b2a8/32.png) [@ThingsTinkerer](https://discourse.nodered.org/u/ThingsTinkerer)\
**Post date:** [30 January 2025 11:34 UTC](https://discourse.nodered.org/t/complexity-of-join-simplify-flow/94992/5 "2025-01-30T11:34:02Z")

</div>

The databases are entirely transparent to us. What is done here is send a single http POST request to thingsboard cloud. They have limits on API usage, theoretically it is 50 msg/s, don't think we ever came close to that. But I have gotten 429 (too many messages) on one occasion. Sending a single message would also reduce network usage, which is on metered connection. Regardless, I don't see any difference in thingsboard whether we send individual messages or grouped together like it is done here.

Last time I worked with statistics using mysql, we used data warehouse-like structure with large fact tables. Each individual datapoint was a column and each row had a shared timestamp. But it doesn't look like that structure is used in thingsboard. I think each data point is handled individually even if sent together.

Finally, while it's not so important to join here, some times I have to work on output of such data as shown here to do further calculations (totals of multiple sources). And then it is a lot easier to work with data nicely grouped together in a single object instead of tying together all the pieces afterwards.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [30 April 2025 11:34 UTC](https://discourse.nodered.org/t/complexity-of-join-simplify-flow/94992/6 "2025-04-30T11:34:13Z")

</div>

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.