# \[Fun Exercise\]Make an MQTT broker in Node-RED

**URL:** <https://discourse.nodered.org/t/fun-exercise-make-an-mqtt-broker-in-node-red/82229>\
**Category:** General\
**Created:** [21 October 2023 08:43 UTC](https://discourse.nodered.org/t/fun-exercise-make-an-mqtt-broker-in-node-red/82229 "2023-10-21T08:43:31Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![cymplecy](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/cymplecy/32/2773_2.png) [@cymplecy](https://discourse.nodered.org/u/cymplecy)\
**Post date:** [21 October 2023 08:43 UTC](https://discourse.nodered.org/t/fun-exercise-make-an-mqtt-broker-in-node-red/82229/1 "2023-10-21T08:43:31Z")

</div>

For fun, I thought - let's build an MQTT broker in Node-RED! 🙂

Anyway, I soon realised that I didn't know how to setup the TCP nodes properly - I often get confused by them ☹

Can someone clue me in on what I should stick in the TCP in/out nodes (or whether I should be using the TCP request one instead?)

Image of my initial flow idea (plan on using port 2883)

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/4/d/4d2400c762b3ee29850b97ed62d6dd0abbb8f5ea.png)

---

<div class="post-metadata">

**Author:** ![marcus-j-davies](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/marcus-j-davies/32/103435_2.png) [@marcus-j-davies](https://discourse.nodered.org/u/marcus-j-davies)\
**Post date:** [21 October 2023 10:13 UTC](https://discourse.nodered.org/t/fun-exercise-make-an-mqtt-broker-in-node-red/82229/2 "2023-10-21T10:13:46Z")

</div>

Hi @cymplecy

Ambitions exercise I must say - I love it! 🤓  
I won't attempt at trying to understand the MQTT protocol itself (I barely use it)

But...  
As a TCP server, the `TCP IN` Node should be `Listen On`  
and I would output it as a `Stream Of` `Buffer` payloads - this is important as some MQTT protocal packets may not be printable ASCII.

As for the `TCP Out` node - this should be `Reply to TCP`

But this will need some careful design work.  
The `TCP IN` Node will include a `_session` object in all output - and the `TCP Out` node will use that, in order to know what connected client the reply should go to.

So if an MQTT Client expects a reply to a command the `_session` that will be included can be left untouched.

But if an MQTT Client sends a command, and by design the protocol requires the result be sent to all clients you need to delete `_session` - so the `TCP Out` will send to all.

But at the same time a command may only be destined for specifc client, and not necessarily, by the Client that sent the command - therefore you need to replace `_session` with the one that belongs to the target client.

You can obtain `_session` for each connected client, using the `status` node on the `connect` event, as that will include the session data `msg.status._session`

I hope that helps

---

<div class="post-metadata">

**Author:** ![cymplecy](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/cymplecy/32/2773_2.png) [@cymplecy](https://discourse.nodered.org/u/cymplecy)\
**Post date:** [21 October 2023 13:01 UTC](https://discourse.nodered.org/t/fun-exercise-make-an-mqtt-broker-in-node-red/82229/3 "2023-10-21T13:01:44Z")

</div>

Thanks for the info 🙂  
I have wormsign!

MQTT out node says connected to my DIY broker (All it does at moment is send a [32,2,0,0] CONNACK packet but at least the MQTT out node thinks its connected to a real broker 🙂

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/e/d/ed9e97bdd53ffa8a9e71d9493ef1cac42995b8b3.png)

---

<div class="post-metadata">

**Author:** ![marcus-j-davies](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/marcus-j-davies/32/103435_2.png) [@marcus-j-davies](https://discourse.nodered.org/u/marcus-j-davies)\
**Post date:** [21 October 2023 16:44 UTC](https://discourse.nodered.org/t/fun-exercise-make-an-mqtt-broker-in-node-red/82229/4 "2023-10-21T16:44:45Z")

</div>

Amazing!

I remember when I studying the ZWave protocal / 100's of specification PDF's to aid in my ZWave Module dev - Great Times!

Cool project to be able to flow out a broker! nice exercise indeed!

keep us updated!

---

<div class="post-metadata">

**Author:** ![cymplecy](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/cymplecy/32/2773_2.png) [@cymplecy](https://discourse.nodered.org/u/cymplecy)\
**Post date:** [21 October 2023 17:06 UTC](https://discourse.nodered.org/t/fun-exercise-make-an-mqtt-broker-in-node-red/82229/5 "2023-10-21T17:06:28Z")

</div>

Got his far - the MQTT node at top is being used to connect/publish to it

So connect works and is acknowledged

Then a ping is received (I've set keepalive time to 15 secs) and acknowledged

Then I publish a message and it received and detected as QoS 0 (so no response is required)

Now need to work on extracting topic and payload from the message and then probably just going to store them in context somewhere

---

<div class="post-metadata">

**Author:** ![marcus-j-davies](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/marcus-j-davies/32/103435_2.png) [@marcus-j-davies](https://discourse.nodered.org/u/marcus-j-davies)\
**Post date:** [21 October 2023 17:19 UTC](https://discourse.nodered.org/t/fun-exercise-make-an-mqtt-broker-in-node-red/82229/6 "2023-10-21T17:19:43Z")

</div>

I don't know the protocol.

But doesn't Clients provide an MQTT client ID?  
if so - you can create a lookup - which will be vital later

This is maybe a little far ahead, but as an example (`global.clients`)

```auto
[
     {clientId: 'xxxxxx', session: <TCP Session Object>},
     {clientId: 'xxxxxx', session: <TCP Session Object>},
     {clientId: 'xxxxxx', session: <TCP Session Object>}
]

```

Then...  
When you know a message is destined for a certain client

```auto
msg._session = global.get('clients').find((C) => C.clientId === <someId>).session
// then send to TCP Reply

```

---

<div class="post-metadata">

**Author:** ![cymplecy](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/cymplecy/32/2773_2.png) [@cymplecy](https://discourse.nodered.org/u/cymplecy)\
**Post date:** [21 October 2023 17:25 UTC](https://discourse.nodered.org/t/fun-exercise-make-an-mqtt-broker-in-node-red/82229/7 "2023-10-21T17:25:07Z")

</div>

Ta  
I'm going to need that down the line 🙂

Initial roadmap is to store the message using topic as key and payload as value

Next will be to try and subscribe to that topic and get my broker to remember the subscription and send it any messages on that topic

---

<div class="post-metadata">

**Author:** ![marcus-j-davies](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/marcus-j-davies/32/103435_2.png) [@marcus-j-davies](https://discourse.nodered.org/u/marcus-j-davies)\
**Post date:** [21 October 2023 18:11 UTC](https://discourse.nodered.org/t/fun-exercise-make-an-mqtt-broker-in-node-red/82229/8 "2023-10-21T18:11:10Z")

</div>

> [@cymplecy](#):
>
> Next will be to try and subscribe to that topic and get my broker to remember the subscription and send it any messages on that topic

In theory should be trivial (so he says not knowing the protocal 😂 )

on `SUBSCRIBE` - using a `global`

```auto
const Subs = global.get('subscriptions')

if(!Subs[<topic>]){
    Subs[<topic>] = [];
}
Subs[<TOPIC>].push(<ClientID>)

```

Then on a message being sent to a topic (we will use our `clients` global for this)

```auto
const Subs = global.get('subscriptions')
const TargetClients = Subs[<TOPIC>]

for(let i = 0;i< TargetClients.length;i++){
   const ClientSession = global.get("clients").find((C) => C.clientId === TargetClients[I]).session
   // To TCP Reply
   node.send({
    _session: ClientSession,
    payload: msg.payload
   })
}

```

---

<div class="post-metadata">

**Author:** ![cymplecy](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/cymplecy/32/2773_2.png) [@cymplecy](https://discourse.nodered.org/u/cymplecy)\
**Post date:** [23 October 2023 08:26 UTC](https://discourse.nodered.org/t/fun-exercise-make-an-mqtt-broker-in-node-red/82229/9 "2023-10-23T08:26:32Z")

</div>

Having spent all weekend trying out using JSONata and contrib nodes to parse and extract data from messages, I've decided life is too short and I'm going to have to write some proper code ☹

Of course, by proper code, I mean use the Blockly node 🙂

---

<div class="post-metadata">

**Author:** ![E1cid](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/e1cid/32/77971_2.png) [@E1cid](https://discourse.nodered.org/u/E1cid)\
**Post date:** [23 October 2023 08:30 UTC](https://discourse.nodered.org/t/fun-exercise-make-an-mqtt-broker-in-node-red/82229/10 "2023-10-23T08:30:16Z")

</div>

Maybe if you posted the data and an example of what you want to parse out of it, someone may offer a solution.

---

<div class="post-metadata">

**Author:** ![cymplecy](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/cymplecy/32/2773_2.png) [@cymplecy](https://discourse.nodered.org/u/cymplecy)\
**Post date:** [23 October 2023 08:46 UTC](https://discourse.nodered.org/t/fun-exercise-make-an-mqtt-broker-in-node-red/82229/11 "2023-10-23T08:46:41Z")

</div>

I'm sure you'd be able to come up with something but that would negate my goal of this being a fun exercise for me. 😄  
Your help in the other thread at least showed me that JSONata has possibilities of parsing it but I just struggle with using it beyond simple stuff ☹

But, if you want some fun yourself 🙂 consider a buffer

[130,9,220,88,0,4,116,101,115,116,2]  
arriving at a change node - the variable text "test" always starts at byte 6, the length (16bit be encoded) is in bytes 4 and 5

So we need to extract the length and then extract the text string

---

<div class="post-metadata">

**Author:** ![E1cid](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/e1cid/32/77971_2.png) [@E1cid](https://discourse.nodered.org/u/E1cid)\
**Post date:** [23 October 2023 11:08 UTC](https://discourse.nodered.org/t/fun-exercise-make-an-mqtt-broker-in-node-red/82229/12 "2023-10-23T11:08:20Z")

</div>

I am not into reinventing the wheel, so best to use a language best suited. Blockly may do it if you can slice and spread the buffer and convert sting.fromCharCode(). JS in a function may be simpler. I know you have an aversion to function nodes, so good luck with Blockly

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [23 October 2023 11:36 UTC](https://discourse.nodered.org/t/fun-exercise-make-an-mqtt-broker-in-node-red/82229/13 "2023-10-23T11:36:45Z")

</div>

As the length is variable, you use buffer parser to output 2 things (in object mode)

1. Extract The 16bit length
2. The rest of the buffer (set length -1, start byte 6)

Then pass that to a function or JSONata or whatever to take `msg.payload.length` bytes from `msg.payload.buffer` and convert it to string.

```auto
msg.payload = msg.payload.slice(msg.payload.length).toString()
return msg

```

_☝untested, written on phone by the beach, while listening to manic street preachers 😉_

  

In a future version of buffer parser I intend on letting users chose `msg.xyz` for the start/length/type etc. to do that currently is rather complex (need to send full specification to buffer parser as an object)

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [23 October 2023 11:51 UTC](https://discourse.nodered.org/t/fun-exercise-make-an-mqtt-broker-in-node-red/82229/14 "2023-10-23T11:51:08Z")

</div>

Your probably want to use .toString('binary') if you don't want to let utf-8 mess thing up... but then again you may not, depending on how you encoded it in the first place... mqtt payloads are just bytes so could be encoded strings or not.

---

<div class="post-metadata">

**Author:** ![jmorris644](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/jmorris644/32/31964_2.png) [@jmorris644](https://discourse.nodered.org/u/jmorris644)\
**Post date:** [23 October 2023 12:58 UTC](https://discourse.nodered.org/t/fun-exercise-make-an-mqtt-broker-in-node-red/82229/15 "2023-10-23T12:58:20Z")

</div>

@cymplecy I know that you started all of this as a fun project. But I am very interested too.

Also, if you search for MQTT in the Manage Pallet there are a number of nodes that have been created. Some of them parse the MQTT messages. You can probably find some methods to use.

---

<div class="post-metadata">

**Author:** ![cymplecy](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/cymplecy/32/2773_2.png) [@cymplecy](https://discourse.nodered.org/u/cymplecy)\
**Post date:** [23 October 2023 13:54 UTC](https://discourse.nodered.org/t/fun-exercise-make-an-mqtt-broker-in-node-red/82229/16 "2023-10-23T13:54:14Z")

</div>

I'll be watching out for utf-8 issues on binary payloads don't you worry 🙂

I've been caught out before when I was developing the [Snap!](https://snap.berkeley.edu) MQTT client extension

---

<div class="post-metadata">

**Author:** ![cymplecy](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/cymplecy/32/2773_2.png) [@cymplecy](https://discourse.nodered.org/u/cymplecy)\
**Post date:** [23 October 2023 13:58 UTC](https://discourse.nodered.org/t/fun-exercise-make-an-mqtt-broker-in-node-red/82229/17 "2023-10-23T13:58:26Z")

</div>

I don't know if I'll carry it on past simple ability to accept a published message from one client and forward onto a single other subscribed client 🙂

---

<div class="post-metadata">

**Author:** ![cymplecy](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/cymplecy/32/2773_2.png) [@cymplecy](https://discourse.nodered.org/u/cymplecy)\
**Post date:** [23 October 2023 14:13 UTC](https://discourse.nodered.org/t/fun-exercise-make-an-mqtt-broker-in-node-red/82229/18 "2023-10-23T14:13:07Z")

</div>

Yep - I played about trying it that way but couldn't work out how to pass in the parameters.

One thing that I think is missing from NR is an easy, none-code way of slicing so well worth (your 🙂 ) effort to make buffer-parser do it 🙂

---

<div class="post-metadata">

**Author:** ![gregorius](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/gregorius/32/73816_2.png) [@gregorius](https://discourse.nodered.org/u/gregorius)\
**Post date:** [23 October 2023 14:17 UTC](https://discourse.nodered.org/t/fun-exercise-make-an-mqtt-broker-in-node-red/82229/19 "2023-10-23T14:17:10Z")

</div>

@cymplecy are you planing on publishing your work? This sounds super interesting for lazy people like me who don't want to setup mosquito (or some other mqtt broker) but still want to have multiple instances of Node-RED communicating with one another.

---

<div class="post-metadata">

**Author:** ![cymplecy](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/cymplecy/32/2773_2.png) [@cymplecy](https://discourse.nodered.org/u/cymplecy)\
**Post date:** [23 October 2023 14:24 UTC](https://discourse.nodered.org/t/fun-exercise-make-an-mqtt-broker-in-node-red/82229/20 "2023-10-23T14:24:49Z")

</div>

> [@gregorius](#):
>
> are you planing on publishing your work?

Yes - once it actually stores and forwards one message 🙂

But it's not for real use

The node-red-contrib-aedes node does a lovely job as a simple pop-up broker (as long as you don't need retained messages)

[Next page](https://discourse.nodered.org/t/fun-exercise-make-an-mqtt-broker-in-node-red/82229.md?page=2)