[Fun Exercise]Make an MQTT broker in Node-RED

General
1

For fun, I thought - let's build an MQTT broker in Node-RED! :slight_smile:

Anyway, I soon realised that I didn't know how to setup the TCP nodes properly - I often get confused by them :frowning:

Can someone clue me in on what I should stick in the TCP in/out nodes (or whether I should be using the TCP request one instead?)

Image of my initial flow idea (plan on using port 2883)

image
image966×236 15.7 KB

2 Likes
2

Hi @cymplecy

Ambitions exercise I must say - I love it! :nerd_face:
I won't attempt at trying to understand the MQTT protocol itself (I barely use it)

But...
As a TCP server, the TCP IN Node should be Listen On
and I would output it as a Stream Of Buffer payloads - this is important as some MQTT protocal packets may not be printable ASCII.

As for the TCP Out node - this should be Reply to TCP

But this will need some careful design work.
The TCP IN Node will include a _session object in all output - and the TCP Out node will use that, in order to know what connected client the reply should go to.

So if an MQTT Client expects a reply to a command the _session that will be included can be left untouched.

But if an MQTT Client sends a command, and by design the protocol requires the result be sent to all clients you need to delete _session - so the TCP Out will send to all.

But at the same time a command may only be destined for specifc client, and not necessarily, by the Client that sent the command - therefore you need to replace _session with the one that belongs to the target client.

You can obtain _session for each connected client, using the status node on the connect event, as that will include the session data msg.status._session

I hope that helps

2 Likes
3

Thanks for the info :slight_smile:
I have wormsign!

MQTT out node says connected to my DIY broker (All it does at moment is send a [32,2,0,0] CONNACK packet but at least the MQTT out node thinks its connected to a real broker :slight_smile:

image
image981×235 14.3 KB

1 Like
4

Amazing!

I remember when I studying the ZWave protocal / 100's of specification PDF's to aid in my ZWave Module dev - Great Times!

Cool project to be able to flow out a broker! nice exercise indeed!

keep us updated!

5

Got his far - the MQTT node at top is being used to connect/publish to it

So connect works and is acknowledged

Then a ping is received (I've set keepalive time to 15 secs) and acknowledged

Then I publish a message and it received and detected as QoS 0 (so no response is required)

Now need to work on extracting topic and payload from the message and then probably just going to store them in context somewhere

6

I don't know the protocol.

But doesn't Clients provide an MQTT client ID?
if so - you can create a lookup - which will be vital later

This is maybe a little far ahead, but as an example (global.clients)

[
     {clientId: 'xxxxxx', session: <TCP Session Object>},
     {clientId: 'xxxxxx', session: <TCP Session Object>},
     {clientId: 'xxxxxx', session: <TCP Session Object>}
]

Then...
When you know a message is destined for a certain client

msg._session = global.get('clients').find((C) => C.clientId === <someId>).session
// then send to TCP Reply
7

Ta
I'm going to need that down the line :slight_smile:

Initial roadmap is to store the message using topic as key and payload as value

Next will be to try and subscribe to that topic and get my broker to remember the subscription and send it any messages on that topic

8

In theory should be trivial (so he says not knowing the protocal :joy: )

on SUBSCRIBE - using a global

const Subs = global.get('subscriptions')

if(!Subs[<topic>]){
    Subs[<topic>] = [];
}
Subs[<TOPIC>].push(<ClientID>)

Then on a message being sent to a topic (we will use our clients global for this)

const Subs = global.get('subscriptions')
const TargetClients = Subs[<TOPIC>]

for(let i = 0;i< TargetClients.length;i++){
   const ClientSession = global.get("clients").find((C) => C.clientId === TargetClients[I]).session
   // To TCP Reply
   node.send({
    _session: ClientSession,
    payload: msg.payload
   })
}
9

Having spent all weekend trying out using JSONata and contrib nodes to parse and extract data from messages, I've decided life is too short and I'm going to have to write some proper code :frowning:

Of course, by proper code, I mean use the Blockly node :slight_smile:

10

Maybe if you posted the data and an example of what you want to parse out of it, someone may offer a solution.

11

I'm sure you'd be able to come up with something but that would negate my goal of this being a fun exercise for me. :smile:
Your help in the other thread at least showed me that JSONata has possibilities of parsing it but I just struggle with using it beyond simple stuff :frowning:

But, if you want some fun yourself :slight_smile: consider a buffer

[130,9,220,88,0,4,116,101,115,116,2]
arriving at a change node - the variable text "test" always starts at byte 6, the length (16bit be encoded) is in bytes 4 and 5

So we need to extract the length and then extract the text string

12

I am not into reinventing the wheel, so best to use a language best suited. Blockly may do it if you can slice and spread the buffer and convert sting.fromCharCode(). JS in a function may be simpler. I know you have an aversion to function nodes, so good luck with Blockly

1 Like
13

As the length is variable, you use buffer parser to output 2 things (in object mode)

  1. Extract The 16bit length
  2. The rest of the buffer (set length -1, start byte 6)

Then pass that to a function or JSONata or whatever to take msg.payload.length bytes from msg.payload.buffer and convert it to string.

msg.payload = msg.payload.slice(msg.payload.length).toString()
return msg

:point_up:untested, written on phone by the beach, while listening to manic street preachers :wink:


In a future version of buffer parser I intend on letting users chose msg.xyz for the start/length/type etc. to do that currently is rather complex (need to send full specification to buffer parser as an object)

1 Like
14

Your probably want to use .toString('binary') if you don't want to let utf-8 mess thing up... but then again you may not, depending on how you encoded it in the first place... mqtt payloads are just bytes so could be encoded strings or not.

2 Likes
15

@cymplecy I know that you started all of this as a fun project. But I am very interested too.

Also, if you search for MQTT in the Manage Pallet there are a number of nodes that have been created. Some of them parse the MQTT messages. You can probably find some methods to use.

16

I'll be watching out for utf-8 issues on binary payloads don't you worry :slight_smile:

I've been caught out before when I was developing the Snap! MQTT client extension

17

I don't know if I'll carry it on past simple ability to accept a published message from one client and forward onto a single other subscribed client :slight_smile:

1 Like
18

Yep - I played about trying it that way but couldn't work out how to pass in the parameters.

One thing that I think is missing from NR is an easy, none-code way of slicing so well worth (your :slight_smile: ) effort to make buffer-parser do it :slight_smile:

19

@cymplecy are you planing on publishing your work? This sounds super interesting for lazy people like me who don't want to setup mosquito (or some other mqtt broker) but still want to have multiple instances of Node-RED communicating with one another.

20

Yes - once it actually stores and forwards one message :slight_smile:

But it's not for real use

The node-red-contrib-aedes node does a lovely job as a simple pop-up broker (as long as you don't need retained messages)