🕸️ MQTT v5 Viewer - View a range of topics with values and extended MQTT v5 data

I will publish this as a Flow once I'm happy that it is a bit more complete. However, as some have expressed an interest ...

A thread was raised on the forum recently about being able to track who/what updated an MQTT topic. It was suggested that MQTT v5 userProperties would be good for this. However, there really doesn't seem to be a decent viewer that includes MQTT v5. MQTT Explorer is kind of the gold standard viewer but it isn't being developed any more. So for some time I've been considering creating something in Node-RED. Here it is. :slight_smile:

This is not a complete example. Current things that need finishing:

  • Live updates - so far the data is received but isn't yet processed.
  • Tidy up the view, highlight the currently selected entry in the tree
  • Add input fields for sending topic/value to MQTT
  • (Allow for data from multiple MQTT brokers) A nice-to-have.

But this already demonstrates the basic concept and is also an example of using Node-RED and UIBUILDER to quickly build data-driven applications with minimal coding.

The flow is pretty simple. you listen for whatever topics you want to monitor. The function node consolidates all of that into a cache object. It also passes the individual inputs on to the front-end for live processing. When a client connects to the UIBUILDER provided endpoint, a cache replay request is automatically sent (built-in feature of UIBUILDER), this triggers the change node to send the full cache. Everything else happens in the front-end but this means that whenever you open the web page (or if you come back to it after some time), you will get the latest full data - this is actually a big step up from MQTT Explorer.

At the moment, you have to reload the page to get the latest data, the live updates haven't been added in the front-end yet but they will be.

[{"id":"c05abcadc6c34fc4","type":"function","z":"b9112384060fcf7c","name":"Cache Entries","func":"// Just a safety measure to prevent too much recursion\n// Change if you have crazy deep topic hierarchies\nconst maxLevels = 9\n\nconst now = new Date()\n\nfunction doTopic(cachePart, keys, level =  0) {\n    if (level > maxLevels) return // no infinite loops!\n\n    let key\n    if (Array.isArray(keys) && keys.length > 1) {\n        // Grab the top level key (make sure it is a string)\n        // MQTT topic parts could be number - don't want an accidental array\n        key = `${keys.shift()}`\n\n        // Make sure that the property exists as the next level\n        if (!cachePart[key]) cachePart[key] = { _count: 1, _hasChildren: true }\n        else {\n            cachePart[key]._count++\n            cachePart[key]._hasChildren = true\n        }\n\n        // Recurse to the next level\n        doTopic(cachePart[key], keys, level++)\n    } else {\n        // Must be at the end of the topic chain\n        if (!Array.isArray) keys = [keys] // make sure it is an array\n\n        // MQTT topic parts could be number\n        key = `${keys[0]}` // make sure it is a string\n        \n        if (!cachePart[key]) cachePart[key] = { _count: 0, _hasChildren: false }\n        cachePart[key]._count++\n        cachePart[key]._hasChildren = false\n        cachePart[key]._updated = now\n        cachePart[key]._value = msg.payload\n        cachePart[key]._userProperties = msg.userProperties\n    }\n}\n\nconst cache = flow.get('mqttCache') ?? {}\n\nconst splitTopic = msg.topic.split('/')\n\ndoTopic(cache, splitTopic)\n\n// flow.set('mqttTopics', topics)\nflow.set('mqttCache', cache)\n\n// Don't do this with bazilions of topics!\nreturn msg","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":520,"y":160,"wires":[["417cabccecf1ef73","bc005152d0db16c7"]]},{"id":"417cabccecf1ef73","type":"debug","z":"b9112384060fcf7c","name":"debug 128","active":false,"tosidebar":true,"console":false,"tostatus":true,"complete":"true","targetType":"full","statusVal":"","statusType":"counter","x":735,"y":160,"wires":[],"l":false},{"id":"ef68803e77ba60b9","type":"mqtt in","z":"b9112384060fcf7c","name":"","topic":"test/#","qos":"2","datatype":"auto-detect","broker":"cc1c989613865d14","nl":false,"rap":true,"rh":0,"inputs":0,"x":230,"y":160,"wires":[["c05abcadc6c34fc4","ecd5ad6aca33c29a"]]},{"id":"ecd5ad6aca33c29a","type":"debug","z":"b9112384060fcf7c","name":"debug 129","active":true,"tosidebar":true,"console":false,"tostatus":true,"complete":"true","targetType":"full","statusVal":"","statusType":"counter","x":345,"y":120,"wires":[],"l":false},{"id":"c19a7abbc8d3e708","type":"mqtt out","z":"b9112384060fcf7c","name":"","topic":"test","qos":"1","retain":"false","respTopic":"test/response","contentType":"text/plain","userProps":"{\"source\":\"Node-RED Dev\"}","correl":"correlation-rules","expiry":"90","broker":"cc1c989613865d14","x":570,"y":360,"wires":[]},{"id":"af2bea4a7ba170f4","type":"inject","z":"b9112384060fcf7c","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":340,"y":360,"wires":[["c19a7abbc8d3e708"]]},{"id":"00d8c50a47ddb8b2","type":"mqtt in","z":"b9112384060fcf7c","name":"","topic":"shellies/#","qos":"2","datatype":"auto-detect","broker":"cc1c989613865d14","nl":false,"rap":true,"rh":0,"inputs":0,"x":240,"y":220,"wires":[["c05abcadc6c34fc4"]]},{"id":"f8ca98892d8a1caf","type":"mqtt in","z":"b9112384060fcf7c","name":"","topic":"ESP/#","qos":"2","datatype":"auto-detect","broker":"cc1c989613865d14","nl":false,"rap":true,"rh":0,"inputs":0,"x":230,"y":280,"wires":[["c05abcadc6c34fc4"]]},{"id":"bc005152d0db16c7","type":"uibuilder","z":"b9112384060fcf7c","name":"","topic":"","url":"mqtt-view","fwdInMessages":false,"allowScripts":false,"allowStyles":false,"copyIndex":true,"templateFolder":"esm-blank-client","extTemplate":"","showfolder":false,"reload":true,"sourceFolder":"src","deployedVersion":"6.6.0","showMsgUib":false,"title":"","descr":"","x":830,"y":240,"wires":[[],["20cbce0e2eaed2be","930295354cacf101"]]},{"id":"20cbce0e2eaed2be","type":"debug","z":"b9112384060fcf7c","name":"debug 130","active":false,"tosidebar":true,"console":false,"tostatus":true,"complete":"true","targetType":"full","statusVal":"","statusType":"counter","x":975,"y":240,"wires":[],"l":false},{"id":"b696be3b9a267b68","type":"change","z":"b9112384060fcf7c","name":"Replay Cache","rules":[{"t":"delete","p":"uibuilderCtrl","pt":"msg"},{"t":"set","p":"payload","pt":"msg","to":"mqttCache","tot":"flow","dc":true},{"t":"set","p":"_uib","pt":"msg","to":"{\"cache\": \"REPLAY\"}","tot":"json"}],"action":"","property":"","from":"","to":"","reg":false,"x":600,"y":240,"wires":[["bc005152d0db16c7"]]},{"id":"930295354cacf101","type":"switch","z":"b9112384060fcf7c","name":"","property":"uibuilderCtrl","propertyType":"msg","rules":[{"t":"eq","v":"client connect","vt":"str"}],"checkall":"true","repair":false,"outputs":1,"x":485,"y":240,"wires":[["b696be3b9a267b68"]],"l":false},{"id":"cc1c989613865d14","type":"mqtt-broker","name":"home","broker":"home.knightnet.co.uk","port":"1883","clientid":"","autoConnect":true,"usetls":false,"protocolVersion":"5","keepalive":"60","cleansession":true,"autoUnsubscribe":true,"birthTopic":"services/nrdev","birthQos":"0","birthRetain":"false","birthPayload":"Online","birthMsg":{"userProps":"{\"source\":\"Node-RED Dev\"}","respTopic":""},"closeTopic":"services/nrdev","closeQos":"0","closeRetain":"false","closePayload":"Offline - disconnected","closeMsg":{"userProps":"{\"source\":\"Node-RED Dev\"}","respTopic":""},"willTopic":"services/nrdev","willQos":"0","willRetain":"false","willPayload":"Offline - unexpected","willMsg":{"userProps":"{\"source\":\"Node-RED Dev\"}","respTopic":""},"userProps":"{\"source\":\"Node-RED Dev\"}","sessionExpiry":""}]

To use the flow, you need to adjust for your own broker and topics. And need to update the UIBUILDER front-end code with the following - note that I'm using the ESM version of the uibuilder client library.

index.html

<!doctype html>
<html lang="en"><head>

    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <link rel="icon" href="../uibuilder/images/node-blue.ico">

    <title>MQTT Viewer - Node-RED uibuilder</title>
    <meta name="description" content="Node-RED uibuilder - MQTT Viewer">

    <!-- Your own CSS -->
    <link type="text/css" rel="stylesheet" href="./index.css" media="all">

    <!-- NOTE this MUST be loaded as a MODULE. Also, uib client loaded in index not here -->
    <script type="module" async src="./index.js">/* Your custom code + Imports uibuilder */</script>

</head><body class="uib">
    
    <h1 class="with-subtitle">MQTT Viewer</h1>
    <div role="doc-subtitle">Using the Node-RED and the UIBUILDER ESM library.</div>

    <div id="container">
        <div id="tree"></div>
        <div id="detail"></div>
    </div>

    <div id="more"><!-- '#more' is used as a parent for dynamic HTML content in examples --></div>

</body></html>

index.css

/* Load defaults from `<userDir>/node_modules/node-red-contrib-uibuilder/front-end/uib-brand.min.css`
 * This version auto-adjusts for light/dark browser settings.
 */
@import url("../uibuilder/uib-brand.min.css");

#container {
    display: grid;
    grid-template-rows: 1fr;
    grid-template-columns: 1fr 1fr;

    gap: 1em;
    height: 100%;
}
#treediv details, #treediv details div {
    margin-left: 1em;
}

.topic {
    cursor: pointer;
}

index.js

// @ts-nocheck
/** An MQTT viewer for Node-RED
 * Use a Node-RED flow to both build a full cache from any MQTT topics you are interested in
 * and also send those topics through to the front-end as well.
 * On initial page load, UIBUILDER sends the full cache, stores it in `cache` and calls buildTreeFromCache().
 * That builds the DOM using the uibuilder front-end library low-code UI feature (saves messing with the DOM).
 *
 * Once the display is created from the cache, any further msgs contain individual topic updates.
 * These update the cache and directly update the UI, again using the low-code capability for simplicity.
 *
 * Two global (window) functions are exposed (showSummary, showDetails). These are used as target functions
 * when a user clicks on the tree structure. They update the details pane on the right-hand side.
 *
 * Author: Julian Knight (Totally Information), October 2023
 * License: Apache 2.0
 */

import '../uibuilder/uibuilder.esm.min.js'  // Adds `uibuilder` and `$` to globals

const cache = {}
const uibVer = Number(uibuilder.get('version').slice(0, 3))

/** Return a deep subset of an object based on a string query
 * @param {object} obj The object to search
 * @param {string} query The string query in the form "level1.level2c.level3f"
 * @param {string} [separator] Optional. Default='.' Query string separator.
 * @returns {any|undefined} The found subset
 */
function getNestedProperty(obj, query, separator = '.') {
    return query.split(separator).reduce((acc, current) => {
        return acc ? acc[current] : undefined
    }, obj)
}

window.showSummary = (event) => {
    let id = event.target.id
    if (!id) id = event.target.parentElement.id
    const details = getNestedProperty(cache, id, '/')
    // console.log('showDetails', id, details, event)

    uibuilder.ui({
        '_ui': [{
            'method': 'replace',
            'components': [
                {
                    'type': 'div',
                    'id': 'detaildiv',
                    'parent': '#detail',
                    'slot': `<p>${id}</p><pre class="syntax-highlight">${(uibVer < 6.6 && details === undefined) ? '**undefined**' : uibuilder.syntaxHighlight(details)}</pre>`,
                },
            ]
        }]
    })
}

window.showDetails = (event) => {
    const details = getNestedProperty(cache, event.target.id, '/')
    // console.log('showDetails', event.target.id, details, event)

    uibuilder.ui({
        '_ui': [{
            'method': 'replace',
            'components': [
                {
                    'type': 'div',
                    'id': 'detaildiv',
                    'parent': '#detail',
                    'slot': `<p>${event.target.id}</p><pre class="syntax-highlight">${uibuilder.syntaxHighlight(details)}</pre>`,
                },
            ]
        }]
    })
}

/** Recursive function to build the tree details
 * @param {*} cachePart The subset of the cache to process next
 * @param {*} topicParent Parent part of the topic for use in titles and element ID's
 * @param {*} level Level tracker to prevent infinite loops
 * @returns {string} uib slot string
 */
function doMoreTree(cachePart, topicParent = '', level = 0) {
    if (level > 9) return ''

    let out = ''

    Object.keys(cachePart).sort().forEach( (key, i) => {
        if (key.startsWith('_') || key === 'updated') return

        const localTopic = `${topicParent}/${key}`.replace(/^\//, '')

        if (cachePart[key]._hasChildren === true) {
            out += `<details id="${localTopic}" title="${localTopic}"><summary class="topic" onclick="showSummary(event)">${key} (${cachePart[key]._count})</summary>${doMoreTree(cachePart[key], localTopic, level++)}</details>`
        } else {
            // End of the tree
            out += `<div id="${localTopic}" class="topic" title="${localTopic}" onclick="showDetails(event)">${key} - ${cachePart[key]._count}</div>`
        }
    })

    return out
}

/** Build the tree view JSON & convert to DOM
 * Replaces any previous entry completely
 * Requires the full MQTT cache which is passed from Node-RED in a msg.payload
 * @param {*} cache The MQTT cache object
 */
function buildTreeFromCache(cache) {
    uibuilder.ui({
        '_ui': [{
            'method': 'replace',
            'components': [
                {
                    'type': 'div',
                    'id': 'treediv',
                    'parent': '#tree',
                    'slot': doMoreTree(cache),
                },
            ]
        }]
    })
}

uibuilder.onChange('msg', (msg) => {
    // if (msg.uibuilderCtrl) console.log({msg})
    if (msg._uib && msg._uib.cache === 'REPLAY') {
        // console.log('CACHE', msg)
        buildTreeFromCache(msg.payload)
    } else if (msg.topic === 'test') console.log({ msg })
})
5 Likes

Screen capture from a slightly earlier version showing the interactivity.

Animation1

2 Likes

You forgot MQTT v5 :stuck_out_tongue: (the reason you resurrected this :joy:

True - I've already included userProperties though :wink:

2 Likes

Looking very good, did you forget the new logo though :wink:

1 Like

Haha, not forgotten. Might be a bit overkill though in this case!

Hmmm, though maybe an animated rotating version .... :rofl:

1 Like

Does this work on uibuilder 6.5.0? I don't have the details window:

Ah, sorry, I forgot to add the extra code. I've already updated 6.6 to cope with a rare edge-case in the syntax highlighting. I think you just need to make sure that you don't pass an undefined entry to the uibuilder.syntaxHighlight(details) entry on line 48 of index.js.


I've put a fix into the index.js code above. The fix grabs the current uib library minor version as a number. Then outputs the string **undefined** instead of the syntax Highlighted version if the version is < 6.6 & details is undefined. Only does that for the showSummary function as that's the only one with the issue.

Yes, now it is working, but it displays "undefined" for everything... I have checked I have details in the cache.

Doh!!! I broke it because I tried to optimise too far. :cry:

Here is the fully working index.js

// @ts-nocheck
/** An MQTT viewer for Node-RED
 * Use a Node-RED flow to both build a full cache from any MQTT topics you are interested in
 * and also send those topics through to the front-end as well.
 * On initial page load, UIBUILDER sends the full cache, stores it in `cache` and calls buildTreeFromCache().
 * That builds the DOM using the uibuilder front-end library low-code UI feature (saves messing with the DOM).
 *
 * Once the display is created from the cache, any further msgs contain individual topic updates.
 * These update the cache and directly update the UI, again using the low-code capability for simplicity.
 *
 * Two global (window) functions are exposed (showSummary, showDetails). These are used as target functions
 * when a user clicks on the tree structure. They update the details pane on the right-hand side.
 *
 * Author: Julian Knight (Totally Information), October 2023
 * License: Apache 2.0
 */

import '../uibuilder/uibuilder.esm.min.js'  // Adds `uibuilder` and `$` to globals

let cache = {}

// Temp fix for highlight can be removed if using uibuilder v6.6 or above
const uibVer = Number(uibuilder.get('version').slice(0, 3))

/** Return a deep subset of an object based on a string query
 * @param {object} obj The object to search
 * @param {string} query The string query in the form "level1.level2c.level3f"
 * @param {string} [separator] Optional. Default='.' Query string separator.
 * @returns {any|undefined} The found subset
 */
function getNestedProperty(obj, query, separator = '.') {
    return query.split(separator).reduce((acc, current) => {
        return acc ? acc[current] : undefined
    }, obj)
}

window.showSummary = (event) => {
    let id = event.target.id
    if (!id) id = event.target.parentElement.id
    const details = getNestedProperty(cache, id, '/')
    // console.log('showDetails', id, details, event)

    uibuilder.ui({
        '_ui': [{
            'method': 'replace',
            'components': [
                {
                    'type': 'div',
                    'id': 'detaildiv',
                    'parent': '#detail',
                    // Temp fix for highlight can be removed if using uibuilder v6.6 or above
                    'slot': `<p>${id}</p><pre class="syntax-highlight">${(uibVer < 6.6 && details === undefined) ? '**undefined**' : uibuilder.syntaxHighlight(details)}</pre>`,
                },
            ]
        }]
    })
}

window.showDetails = (event) => {
    const details = getNestedProperty(cache, event.target.id, '/')
    // console.log('showDetails', event.target.id, details, cache, event)

    uibuilder.ui({
        '_ui': [{
            'method': 'replace',
            'components': [
                {
                    'type': 'div',
                    'id': 'detaildiv',
                    'parent': '#detail',
                    'slot': `<p>${event.target.id}</p><pre class="syntax-highlight">${uibuilder.syntaxHighlight(details)}</pre>`,
                },
            ]
        }]
    })
}

/** Recursive function to build the tree details
 * @param {*} cachePart The subset of the cache to process next
 * @param {*} topicParent Parent part of the topic for use in titles and element ID's
 * @param {*} level Level tracker to prevent infinite loops
 * @returns {string} uib slot string
 */
function doMoreTree(cachePart, topicParent = '', level = 0) {
    if (level > 9) return ''

    let out = ''

    Object.keys(cachePart).sort().forEach( (key, i) => {
        if (key.startsWith('_') || key === 'updated') return

        const localTopic = `${topicParent}/${key}`.replace(/^\//, '')

        if (cachePart[key]._hasChildren === true) {
            out += `<details id="${localTopic}" title="${localTopic}"><summary class="topic" onclick="showSummary(event)">${key} (${cachePart[key]._count})</summary>${doMoreTree(cachePart[key], localTopic, level++)}</details>`
        } else {
            // End of the tree
            out += `<div id="${localTopic}" class="topic" title="${localTopic}" onclick="showDetails(event)">${key} - ${cachePart[key]._count}</div>`
        }
    })

    return out
}

/** Build the tree view JSON & convert to DOM
 * Replaces any previous entry completely
 * Requires the full MQTT cache which is passed from Node-RED in a msg.payload
 * @param {*} cache The MQTT cache object
 */
function buildTreeFromCache(cache) {
    uibuilder.ui({
        '_ui': [{
            'method': 'replace',
            'components': [
                {
                    'type': 'div',
                    'id': 'treediv',
                    'parent': '#tree',
                    'slot': doMoreTree(cache),
                },
            ]
        }]
    })
}

uibuilder.onChange('msg', (msg) => {
    // if (msg.uibuilderCtrl) console.log({msg})
    if (msg._uib && msg._uib.cache === 'REPLAY') {
        // console.log('CACHE', msg)
        cache = msg.payload
        buildTreeFromCache(cache)
    } else if (msg.topic === 'test') console.log({ msg })
})

Great! Thank you!

1 Like

I'm interested to see how far people push this - can you subscribe to # and $SYS/# and see whether that has problems? Hint: you might want to put a rate limiter node in place and work up to see what the maximum feasible rate is. :slight_smile:

1 Like

And some very quick tests show that the initial build of the tree with a very large set of topics (approximately 13,000) takes no more than around 40ms. Remember, this is building a JSON describing the whole tree and then converting it to actual DOM.

The individual updates are arriving in the front-end at anywhere up to nearly 8,000 times a second. Not yet processing them of course but I'm seeing about 100MB of RAM in use for the tab with CPU up to about 1.2% max, mostly a lot lower. Though admittedly, that is on an i9 mobile processor.

Hmm, and it makes a big difference to CPU to have the dev tools open when receiving large volumes of websocket packets! ~1% without and around 4-5% with.

Oh boy! Just realised how much data Node-RED is slinging with this many MQTT and websocket packets! Around 350-450 Mbps! 16% CPU. I'd say that's pretty impressive.

5 Likes

Thanks Julian!

This works well. Using an i7 laptop and I logged over 200 individual messages from the server. No rate limiter and the CPU is not even noticing the load (<0.06%). Deepest tree is 4 levels and includes Shellies, Tasmota (Sonoff and ESP8266) and the OpenTherm Gateway.

3 Likes

Wow fantastic. It almost looks like MQTT Explorer which is my go-to MQTT viewer. But this is great if you want to check the traffic on a remote system.

1 Like

no idea what superlatives I should use to show my exploding brain-drafts
awesome? insane? ...
Not yet using uibuilder, but I guess this is just a things of days ...
I have to close my browser for today or my day is gone :wink:
This drives my thoughts crazy - will my raspidisplay hold this |-)))

stunning ... my mouth is still open ...

2 Likes

UIBUILDER reduced the development time from days to just a couple of hours or so. :wink:

[Feature Request]
Can there be an option to display binary payloads better than MQTT Explorer does

i.e display
cymplecy/earthnullschool/wind/binary_tile as something like the debug sidebar does
image

I think that depends on being able to discover that a value is not text. Might have to be a manual button.

Yes- just asking for a manual button :slight_smile:

1 Like