Node-RED version of MQTT Explorer

[TotallyInformation]

Hi all, In a recent thread, I promised @Trying_to_learn that I would revisit a project I started quite a long time ago now, which is a replacement for the MQTT Explorer desktop app.

So here is my first draft. It is still very basic but it demonstrates the principles. If you have anything you particularly want to see, let me know. Not promising anything but I will certainly add to the backlog.

The idea here is to have a useful, purely web-based view into an MQTT environment. But also to showcase what Node-RED and UIBUILDER can achieve.

All the clever front-end bits are done in the front-end code and Node-RED is simply used to provide the environment, the connector to MQTT and the comms to the front-end. I am just dumping the output from an MQTT-in node straight to UIBUILDER. With a rate limiter thrown in just to be on the safe side.

Eventually, I will doubtless add the ability to send MQTT messages as well but for now, it is just a very simple UI using native HTML, CSS and JavaScript. No front-end frameworks are used or needed.

Please note that I am making use of fairly new CSS features. These all work in all current desktop (and I think mobile - let me know) browsers. I am using the UIBUILDER uib-brand CSS file as the base and I use the built-in uibuilder.sysntaxHighlight(data) function to do some simple but nice formatting of the MQTT payloads. The tree view is native HTML <details> and <summary> tags. The layout uses CSS grid. The details display uses HTML <template> elements that are cloned to the details panel as needed which saves a load of processing and memory storage of data.

I haven't cached the inputs as yet, if you're MQTT broker is anything like mine, you will have a LOT of data flowing through. However, it probably makes sense to cache the latest retained messages which I will do. With UIBUILDER, you can easily use multiple uib-cache nodes to have different strategies for different topics. You might, for example, want to cache a thousand of some critical message but not of others.

I will share the front-end code for this in a follow-up message shortly.

[{"id":"9f1e743afa926865","type":"uibuilder","z":"4393d057899d5287","name":"","topic":"","url":"mqtt-explorer","okToGo":true,"fwdInMessages":false,"allowScripts":false,"allowStyles":false,"copyIndex":true,"templateFolder":"blank","extTemplate":"","showfolder":false,"reload":false,"sourceFolder":"src","deployedVersion":"5.0.0-dev.2","showMsgUib":false,"title":"A Node-RED replacement for MQTT Explorer","descr":"","editurl":"vscode://file/D:\\src\\uibRoot/mqtt-explorer/?windowId=_blank","x":1040,"y":200,"wires":[["7d8dc679995a88bd"],["affb2d9a74d04af4"]]},{"id":"c97df9205436c418","type":"mqtt in","z":"4393d057899d5287","name":"","topic":"#","qos":"2","datatype":"auto","broker":"1fe2972256cb8a87","nl":false,"rap":true,"rh":0,"inputs":0,"x":190,"y":180,"wires":[["9bc3728ee0f43c66"]]},{"id":"ae90065eb89d6ad5","type":"switch","z":"4393d057899d5287","name":"Filter out some very high volume topics","property":"topic","propertyType":"msg","rules":[{"t":"cont","v":"nrlog","vt":"str"},{"t":"cont","v":"telegraf","vt":"str"},{"t":"else"}],"checkall":"false","repair":false,"outputs":3,"x":570,"y":180,"wires":[[],[],["046efcf288d44964"]]},{"id":"a8081162a034775c","type":"mqtt in","z":"4393d057899d5287","name":"","topic":"$SYS/#","qos":"2","datatype":"auto","broker":"1fe2972256cb8a87","nl":false,"rap":true,"rh":0,"inputs":0,"x":190,"y":240,"wires":[[]]},{"id":"7d8dc679995a88bd","type":"debug","z":"4393d057899d5287","name":"","active":false,"tosidebar":true,"console":false,"tostatus":true,"complete":"true","targetType":"full","statusVal":"","statusType":"counter","x":1175,"y":180,"wires":[],"l":false},{"id":"affb2d9a74d04af4","type":"debug","z":"4393d057899d5287","name":"","active":false,"tosidebar":true,"console":false,"tostatus":true,"complete":"true","targetType":"full","statusVal":"","statusType":"counter","x":1175,"y":220,"wires":[],"l":false},{"id":"046efcf288d44964","type":"delay","z":"4393d057899d5287","name":"","pauseType":"rate","timeout":"5","timeoutUnits":"seconds","rate":"10000","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"allowrate":false,"outputs":1,"x":850,"y":200,"wires":[["9f1e743afa926865"]]},{"id":"9bc3728ee0f43c66","type":"change","z":"4393d057899d5287","name":"Add \\n Timestamp","rules":[{"t":"set","p":"lastUpdate","pt":"msg","to":"","tot":"date"}],"action":"","property":"","from":"","to":"","reg":false,"x":330,"y":180,"wires":[["ae90065eb89d6ad5"]]},{"id":"1fe2972256cb8a87","type":"mqtt-broker","name":"mqtt5-testing","broker":"home.knightnet.co.uk","port":"1883","clientid":"desktop-nr-mqtt5-testing","autoConnect":true,"usetls":false,"protocolVersion":"5","keepalive":"60","cleansession":true,"birthTopic":"DEV/mqtt5-testing","birthQos":"0","birthRetain":"false","birthPayload":"Online","birthMsg":{},"closeTopic":"DEV/mqtt5-testing","closeQos":"0","closeRetain":"false","closePayload":"Offline","closeMsg":{},"willTopic":"DEV/mqtt5-testing","willQos":"0","willRetain":"false","willPayload":"Broken","willMsg":{},"sessionExpiry":""},{"id":"d99e9240e9cdc815","type":"global-config","env":[],"modules":{"node-red-contrib-uibuilder":"7.6.0"}}]

[TotallyInformation]

Replace the following into the uibuilder node's files - also delete the index.js file as it isn't needed.

index.html

<!doctype html>
<html lang="en"><head>

    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <link rel="icon" href="../uibuilder/images/node-blue.ico">

    <title>MQTT Explorer - Node-RED UIBUILDER</title>
    <meta name="description" content="Node-RED UIBUILDER - MQTT Explorer">

    <!-- Your own CSS (defaults to loading uibuilders css)-->
    <link type="text/css" rel="stylesheet" href="./index.css" media="all">

    <script type="module" src="./index.mjs"></script>

</head><body>
    
    <h1 class="with-subtitle">MQTT Explorer</h1>
    <div role="doc-subtitle">Using UIBUILDER for Node-RED</div>

    <!-- '#more' is used as a parent for dynamic HTML content in examples -->
    <div id="more" uib-topic="more"></div>
    <div id="container">
        <div id="topics">
            <h2>MQTT Topics</h2>
        </div>
        <div id="details">
            <h2>Message Details</h2>
            <div id="detailContent">Click on a topic to see message details here.</div>
        </div>
    </div>

</body></html>

index.mjs

Note the extension - not .js but .mjs - it is a JavaScript ES Module.

Also note the use of HTML templates. These hold HTML but do not display it. JavaScript is used to Clone that content elsewhere as needed where is is shown.

// @ts-nocheck
// Give VS Code IntelliSense for uibuilder
/// <reference path="../types/uibuilder.d.ts" />

// @ts-ignore
import uibuilder from '../uibuilder/uibuilder.esm.min.js'

/** @type {HTMLElement} Get fixed ref to the more div */
const elTopics = document.getElementById('topics')
/** @type {HTMLElement} Get fixed ref to the details div */
const elDetailContent = document.getElementById('detailContent')

// Specify an HTML ID compatible separator string for topics - cannot clash with MQTT topic chars and can only contain _ or -
const topicSeparator = '___'

/** Format and return the details HTML for a given MQTT value input
 * @param {string|object} input The MQTT value input
 * @param {HTMLElement} elCurrent The current details element
 * @param {object} msg Node-RED message object
 * @returns 
 */
function details(input, elCurrent, msg) {
    const lastUpdate = new Date(msg.lastUpdate).toISOString() || new Date().toISOString()
    try {
        input = JSON.parse(input)
    } catch (e) { /* Not JSON, do nothing */ }
    return `
        <br><b>Topic</b>: ${msg.topic}<br>
        <b>Timestamp</b>: ${lastUpdate} | <b>QoS</b>: ${msg.qos} | <b>Retained?</b>: ${msg.retain}
        <pre class="syntax-highlight">${uibuilder.syntaxHighlight(input)}</pre>
    `
}

// Listen for incoming messages from Node-RED and action
uibuilder.onChange('msg', (msg) => {
    console.log({msg})
    // console.group(`MQTT Topic: ${msg.topic}`)

    // We are assuming that the topic levels are separated by '/'
    const splitTopic = msg.topic.split('/')
    const topicLeaf = splitTopic[splitTopic.length - 1]

    // Set these up so they are still in scope after the loop
    let parentId = 'topics'
    let elParent = elTopics
    let idTopic = ''
    let elCurrent = null
    let elSummary = null

    // Loop through each topic level to build out the nested structure
    const finalLevel = splitTopic.length - 1
    splitTopic.forEach( (topic, level) => {
        if (level > 0) {
            parentId = splitTopic.slice(0, level).join(topicSeparator)
            elParent = document.getElementById(parentId)
            idTopic = `${parentId}${topicSeparator}${topic}`
        } else {
            idTopic = topic
        }
        if (!elParent) { // Sanity check, should not happen
            console.warn(`Parent element with ID ${parentId} not found!`)
            return
        }
        
        elCurrent = document.getElementById(idTopic)
        // Current topic doesn't exist in the DOM, so create a new child element
        if (!elCurrent) {
            elCurrent = document.createElement('details')
            elSummary = document.createElement('summary')
            elCurrent.id = idTopic
            elSummary.textContent = topic
            elCurrent.appendChild(elSummary)
            elParent.appendChild(elCurrent)
        }
    })

    /** @type {HTMLTemplateElement} Does the final topic level already have a data template? */
    let elTemplate = elCurrent.querySelector('template')
    // If not, add one now and keep a reference to it
    if (!elTemplate) {
        elTemplate = document.createElement('template')
        elCurrent.appendChild(elTemplate)
    }
    // Add/Update the current timestamp to the data-last-update attribute
    elCurrent.dataset.lastUpdate = new Date().toISOString()
    if (msg.payload !== undefined) {
        // Add the hasData class to the final topic level
        elCurrent.classList.add('hasData')

        // Add the message payload to the final topic level template
        const elPayload = document.createElement('div')
        elPayload.innerHTML = details(msg.payload, elCurrent, msg)
        elTemplate.content.appendChild(elPayload)
    } else {
        // No payload, so just ensure the hasData class is not present
        elCurrent.classList.remove('hasData')
        // and remove any existing payload element
        const elPayloadCheck = elCurrent.querySelector('div')
        if (elPayloadCheck) {
            elCurrent.removeChild(elPayloadCheck)
        }
    }
    // Add a click event to the any section that has the hasData class that clones the template content to the details div
    if (elCurrent.classList.contains('hasData') && elSummary) {
        elSummary.onclick = (event) => {
            // don't propagate to parent details elements
            event.stopPropagation()
            elDetailContent.innerHTML = '' // Clear existing content
            const clone = elTemplate.content.cloneNode(true)
            // console.log(`Clicked on topic: ${msg.topic}`, {event, elTemplate, clone, elDetails: elDetailContent})
            elDetailContent.appendChild(clone)
        }
    }

    // console.groupEnd()
})

index.css

Note the use of nested definitions and other modern CSS.

/* Load defaults from `<userDir>/node_modules/node-red-contrib-uibuilder/front-end/uib-brand.min.css`
 * This is optional but reasonably complete and allows for light/dark mode switching.
 */
@import url("../uibuilder/uib-brand.min.css");

/* for any element that has an id that contains ___ use the following style */
[id*="___"] {
    margin-left: 1em;
}

details {
    grid-area: topics;

    &:not(.hasData) > summary {
        font-weight: bold;
    }
    &.hasData > summary {
        font-style: italic;
    }
}

/* Override the uib-brand std height of 22em */
.syntax-highlight {
    height: auto;
}

/* Format #more as a grid using 2 columns with named areas */
#container {
    display: grid;
    gap: 1em;
    /* grid-template-columns: 1fr 1fr; */
    grid-template-areas:
        "topics details";
}

#topics {
    grid-area: topics;
    border-right: 1px solid var(--text1);

    /* If a topics>details has no details inside, make the summary inline rather than block */
    details:not(:has(details)) > summary {
        display: inline;

        /* and hide the disclosure triangle */
        &::-webkit-details-marker {
            display: none;
        }
    }
}

#details {
    grid-area: details;
    /* padding-left: 1em; */
}

[marcus-j-davies]

Awesome amount of work Julian!
Very cool achievement

Whilst I don't use MQTT (Much) - showing the interconnectivity between flow data and the UI - very cool

[marcus-j-davies]

And Pssttt...

Combine this with SFE = A web based MQTT explorer powered by Node RED as an executable

[TotallyInformation]

@marcus-j-davies

as an executable

I have too many apps installed already! Just another thing to maintain. With this, I'm already maintaining node-red and uibuilder but that is all. Well, and my browser I suppose. But this gives access from anywhere that I can access my Node-RED instance over HTTP(S). With my (nearly complete) write-up of setting up Cloudflare Zero Trust, this could easily and safely be accessed from anywhere in the world (well, except those countries that are in my ban list!).

Of course, this is all open and free so anyone can create an executable version as they please.

[TotallyInformation]

Here is a quick to-do list for the project:

In the Front End

• Add visual indicator to topics with data.
• Add copy ability to topic names and values.
• Add text (maybe truncated) to topics with values.
• Add animation when new data arrives.
• Add variable to restrict the number of kept messages per topic level.
• Add search/filtering of topics.
• Add ability to send a new msg to a topic.
• Add ability to show other metadata (e.g. MQTT v5 properties).
• Add charting of numeric data over time.
• Add msg count and sub-topic count per topic level.
• Add ability to remove topics or clear data.
• Add ability to export data (e.g. JSON, CSV).
• Highlight currently selected topic.
• Colour-code retained messages in the topics list.

In Node-RED

• Add caching.
• Allow dynamic changing of subscriptions.

There will be more, I've no doubt.

[marcus-j-davies]

@TotallyInformation,

Is the flow and the 3 files - the full set of deps to get this running?
I will compile a quick SFE (Windows) if so - just for kicks.

EDIT:
I should note; I'm not hijacking your achievement - as its for my own testing - but happy to share the SFE if you wish to demo UIB, else its for my own testing (privately if you prefer)

[TotallyInformation]

@marcus-j-davies

Is the flow and the 3 files - the full set of deps to get this running?

Well, your flow needs to connect to your MQTT broker and not mine of course.

But otherwise, yes.

@marcus-j-davies

I should note; I'm not hijacking your achievement

No worries, hijack away. And please do share, more than happy for this to spread.

[marcus-j-davies]

I'll compile an SFE (Windows) - that auto loads the front end upon launch, and will share the download on a PM - feel free to share (or advise me you are happy for me to post it here)

Having a Node RED/UIB Powered MQTT explorer compliments my Gin on a Friday

[TotallyInformation]

@marcus-j-davies

or advise me you are happy for me to post it here

Yes, of course, go right ahead.

[Trying_to_learn]

You aren't being paid enough.

This sounds like a great project.

[thalesmaoa]

For me, MQTT Explorer is one of the best tools for development. It is lightweight, multi-platform, simple, and easy to use.
Improving it would be awesome. Here are some things that I miss the most:
Filter capabilities for payload data. Especially for big JSON messages.
Scripting support like MQTTX. Sometimes you just want to simply add data or check conditions.
MQTT 5.0 properties view.

[marcus-j-davies]

@TotallyInformation

UIB/Node RED Powered MQTT Explorer in an SFE.
AUTOLOAD file just fires up the default browser, and points to /mqtt-explorer

Though, your admin API v3 is bringing in node:inspector, which was not used, so had to remove it, as currently compiled Node App binaries don't have debug support

nodes/libs/admin-api-v3.cjs

[mudwalker]

Had a quick try. this looks good, especially as it can be made accessible remotely as a Web page using whichever preferred method.

[Colin]

Your flow has at the end of it
{"id":"d99e9240e9cdc815","type":"global-config","env":[],"modules":{"node-red-contrib-uibuilder":"7.6.0"}}
So when I import it node-red asks if I want to install that version of uibuilder, but npm only has 7.5.0 so the install fails.
I can install that version manually of course.

[TotallyInformation]

@marcus-j-davies

admin API v3 is bringing in node:inspector

Ah, good spot, I'll have to check that.

@Colin

7.6.0

Oops! Thanks Colin. I did it on my dev PC of course which has the next version of uibuilder in it. My bad. It should all work fine with v7.5 anyway. I didn't realise that the exported flow file includes version numbers.

[Colin]

@TotallyInformation

I didn't realise that the exported flow file includes version numbers.

Neither did I till it failed when I was prompted to install it.

[thebaldgeek]

Just to be clear, this flow requires UIBUILDER?
I see a missing node error, but its not clear if installing this node is all that is required, or more tweaking of other files as per the second post is required.

I've been testing many MQTT Explorers, but never found one that works as cleanly as MQTT itself does. I'd love to test this out if possible.

[mudwalker]

Hi Ben,

Yes, you do need UIBUILDER to run this. I compared it with MQTT Explorer and it does work well. As Julian says, this is an 'early version' and he accepts any suggestions you make and will add them to his backlog.

HTH,
Colin J

[Colin]

A small negative feature. If there is topic a/b that has a value, but also a/b/c has a value then clicking on a/b does not show the value, it just expands that topic allowing you to click on a/b/c, which then shows a/b and a/b/c. I think one should be able to see the value of a/b by clicking it.

In terms of upcoming features, being able to see mqtt5 user properties would be useful to me. I have started adding a user property source to all topics, where the source is the name of the server that is publishing the data. I don't use that in flows but it can be useful to know which server published a message, particularly when debugging. I think I might add a timestamp field too, which could be useful.