Receiving "chunks" of data via MQTT and adding to Dashboard chart

Dashboard
1

I have a datalogger system that is recording multiple data points (and associated timestamps) more quickly than one would want to publish via MQTT, so each MQTT submission includes multiple data that I want to append to a chart on the dashboard. I'm running into two problems right now:

  1. There should be 4 series, and the legend shows 4 series, and the payload I'm sending to chart has data in all 4 series, but I only see data points from the first 3 series on the chart (see chart image below)
  2. The payload sent to the chart is not appending to the data from the previous message, it is overwriting the data from the previous message. Since the data from each MQTT publication contains multiple data in each time series, I'm not adding data one at a time. But from what I've read the only other option is to tell the chart that the payload is meant to be the entire chart. Is there a way to append multiple data points from a payload onto a chart with existing data?

As an example, here is the data as received from MQTT:

{"Ch1":[[3716421,3716479,3716545,3716613,3716680,3716747,3716815,3716883,3716950,3717018,3717085,3717152,3717220,3717288,3717356],[0.000666,0.000662,0.000658,0.000659,0.000663,0.000662,0.000658,0.00066,0.00066,0.000657,0.000662,0.000657,0.000654,0.00065,0.000654]],"Ch2":[[3716426,3716494,3716561,3716629,3716697,3716764,3716832,3716899,3716967,3717034,3717102,3717169,3717237,3717304,3717372],[-0.000449,-0.000452,-0.000457,-0.000454,-0.000458,-0.000461,-0.000458,-0.000462,-0.000466,-0.000463,-0.000461,-0.000463,-0.000464,-0.000461,-0.000459]],"Ch3":[[3716443,3716511,3716578,3716646,3716713,3716781,3716848,3716916,3716984,3717051,3717119,3717186,3717254,3717321,3717389],[0.000359,0.000359,0.000358,0.000358,0.000358,0.000358,0.000356,0.000358,0.000359,0.000359,0.000358,0.000359,0.000361,0.000359,0.000362]],"Ch4":[[3716460,3716528,3716595,3716663,3716730,3716798,3716865,3716933,3717000,3717068,3717135,3717203,3717270,3717338,3717406],[0.002398,0.002397,0.002397,0.002395,0.002396,0.002395,0.002392,0.00239,0.00239,0.002387,0.002387,0.002385,0.002384,0.002384,0.002381]],"ID":"00E5002C"}

(4 channels each with timestamps in millis and data, and then an ID for the logger itself)

A function processes this into the following payload sent to the chart:

[{"series":["Ch1","Ch2","Ch3","Ch4"],"data":[[[{"x":3716421,"y":0.000666}],[{"x":3716479,"y":0.000662}],[{"x":3716545,"y":0.000658}],[{"x":3716613,"y":0.000659}],[{"x":3716680,"y":0.000663}],[{"x":3716747,"y":0.000662}],[{"x":3716815,"y":0.000658}],[{"x":3716883,"y":0.00066}],[{"x":3716950,"y":0.00066}],[{"x":3717018,"y":0.000657}],[{"x":3717085,"y":0.000662}],[{"x":3717152,"y":0.000657}],[{"x":3717220,"y":0.000654}],[{"x":3717288,"y":0.00065}],[{"x":3717356,"y":0.000654}]],[{"x":3716426,"y":-0.000449},{"x":3716494,"y":-0.000452},{"x":3716561,"y":-0.000457},{"x":3716629,"y":-0.000454},{"x":3716697,"y":-0.000458},{"x":3716764,"y":-0.000461},{"x":3716832,"y":-0.000458},{"x":3716899,"y":-0.000462},{"x":3716967,"y":-0.000466},{"x":3717034,"y":-0.000463},{"x":3717102,"y":-0.000461},{"x":3717169,"y":-0.000463},{"x":3717237,"y":-0.000464},{"x":3717304,"y":-0.000461},{"x":3717372,"y":-0.000459}],[{"x":3716443,"y":0.000359},{"x":3716511,"y":0.000359},{"x":3716578,"y":0.000358},{"x":3716646,"y":0.000358},{"x":3716713,"y":0.000358},{"x":3716781,"y":0.000358},{"x":3716848,"y":0.000356},{"x":3716916,"y":0.000358},{"x":3716984,"y":0.000359},{"x":3717051,"y":0.000359},{"x":3717119,"y":0.000358},{"x":3717186,"y":0.000359},{"x":3717254,"y":0.000361},{"x":3717321,"y":0.000359},{"x":3717389,"y":0.000362}],[{"x":3716460,"y":0.002398},{"x":3716528,"y":0.002397},{"x":3716595,"y":0.002397},{"x":3716663,"y":0.002395},{"x":3716730,"y":0.002396},{"x":3716798,"y":0.002395},{"x":3716865,"y":0.002392},{"x":3716933,"y":0.00239},{"x":3717000,"y":0.00239},{"x":3717068,"y":0.002387},{"x":3717135,"y":0.002387},{"x":3717203,"y":0.002385},{"x":3717270,"y":0.002384},{"x":3717338,"y":0.002384},{"x":3717406,"y":0.002381}]],"labels":[""]}]

"Data" does indeed have 4 arrays of 15 object elements each (x/y values)

Here's the flow:

[
    {
        "id": "2d177c76a333981b",
        "type": "tab",
        "label": "Flow 1",
        "disabled": false,
        "info": "",
        "env": []
    },
    {
        "id": "58aecee39ac1fea0",
        "type": "mqtt in",
        "z": "2d177c76a333981b",
        "name": "",
        "topic": "outTopic",
        "qos": "2",
        "datatype": "auto-detect",
        "broker": "3cba6ed55fa3c2fb",
        "nl": false,
        "rap": true,
        "rh": 0,
        "inputs": 0,
        "x": 360,
        "y": 300,
        "wires": [
            [
                "369f80752f62d5ab",
                "03d6ef3d2ba33214"
            ]
        ]
    },
    {
        "id": "369f80752f62d5ab",
        "type": "debug",
        "z": "2d177c76a333981b",
        "name": "debug 1",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "statusVal": "",
        "statusType": "auto",
        "x": 520,
        "y": 380,
        "wires": []
    },
    {
        "id": "346d14cb602666a0",
        "type": "ui_chart",
        "z": "2d177c76a333981b",
        "name": "",
        "group": "bb43d51ed33cfd60",
        "order": 0,
        "width": "6",
        "height": "6",
        "label": "Datalogger 1",
        "chartType": "line",
        "legend": "true",
        "xformat": "HH:mm:ss",
        "interpolate": "linear",
        "nodata": "",
        "dot": true,
        "ymin": "",
        "ymax": "",
        "removeOlder": 1,
        "removeOlderPoints": "",
        "removeOlderUnit": "3600",
        "cutout": 0,
        "useOneColor": false,
        "useUTC": false,
        "colors": [
            "#1f77b4",
            "#aec7e8",
            "#ff7f0e",
            "#2ca02c",
            "#98df8a",
            "#d62728",
            "#ff9896",
            "#9467bd",
            "#c5b0d5"
        ],
        "outputs": 1,
        "useDifferentColor": false,
        "className": "",
        "x": 930,
        "y": 280,
        "wires": [
            [
                "c493d2ea78045827",
                "72d1d41e3df2ae4e"
            ]
        ]
    },
    {
        "id": "09a8a59620e214af",
        "type": "debug",
        "z": "2d177c76a333981b",
        "name": "debug 2",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "statusVal": "",
        "statusType": "auto",
        "x": 860,
        "y": 400,
        "wires": []
    },
    {
        "id": "03d6ef3d2ba33214",
        "type": "function",
        "z": "2d177c76a333981b",
        "name": "function 1",
        "func": "var data = [];\nvar data1 = [];\nvar data2 = [];\nvar data3 = [];\nvar data4 = [];\n\nvar series =[\"Ch1\", \"Ch2\", \"Ch3\", \"Ch4\"];\nvar labels = [\"\"];\n\nfor (var i = 0; i< msg.payload.Ch1[0].length; i++){\n    data1.push([{\"x\": msg.payload.Ch1[0][i], \"y\" : msg.payload.Ch1[1][i]}]);\n}\n\nfor (var i = 0; i< msg.payload.Ch2[0].length; i++){\n    data2.push({\"x\": msg.payload.Ch2[0][i], \"y\" : msg.payload.Ch2[1][i]});\n}\n\nfor (var i = 0; i< msg.payload.Ch3[0].length; i++){\n    data3.push({\"x\": msg.payload.Ch3[0][i], \"y\" : msg.payload.Ch3[1][i]});\n}\n\nfor (var i = 0; i< msg.payload.Ch4[0].length; i++){\n    data4.push({\"x\": msg.payload.Ch4[0][i], \"y\" : msg.payload.Ch4[1][i]});\n}\n\ndata = [data1, data2, data3, data4];\nmsg.payload = [{series, data, labels}];\n\nreturn msg;",
        "outputs": 1,
        "timeout": 0,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 680,
        "y": 260,
        "wires": [
            [
                "09a8a59620e214af",
                "346d14cb602666a0"
            ]
        ]
    },
    {
        "id": "476cd736b2739655",
        "type": "ui_button",
        "z": "2d177c76a333981b",
        "name": "Clear",
        "group": "bb43d51ed33cfd60",
        "order": 1,
        "width": "2",
        "height": "1",
        "passthru": false,
        "label": "Clear chart",
        "tooltip": "",
        "color": "",
        "bgcolor": "",
        "className": "",
        "icon": "",
        "payload": "[]",
        "payloadType": "json",
        "topic": "",
        "topicType": "str",
        "x": 670,
        "y": 140,
        "wires": [
            [
                "346d14cb602666a0"
            ]
        ]
    },
    {
        "id": "c493d2ea78045827",
        "type": "debug",
        "z": "2d177c76a333981b",
        "name": "debug 3",
        "active": false,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "statusVal": "",
        "statusType": "auto",
        "x": 1200,
        "y": 280,
        "wires": []
    },
    {
        "id": "1ad58c6121419e17",
        "type": "ui_button",
        "z": "2d177c76a333981b",
        "name": "",
        "group": "bb43d51ed33cfd60",
        "order": 3,
        "width": "2",
        "height": "1",
        "passthru": false,
        "label": "Save Charts",
        "tooltip": "",
        "color": "",
        "bgcolor": "",
        "className": "",
        "icon": "",
        "payload": "",
        "payloadType": "str",
        "topic": "save",
        "topicType": "str",
        "x": 690,
        "y": 500,
        "wires": [
            [
                "72d1d41e3df2ae4e"
            ]
        ]
    },
    {
        "id": "72d1d41e3df2ae4e",
        "type": "function",
        "z": "2d177c76a333981b",
        "name": "",
        "func": "if (msg.topic === \"save\") {\n    msg.payload = context.last;\n    return msg;\n}\nelse {\n    context.last = msg.payload;\n}\nreturn null;",
        "outputs": 1,
        "timeout": "",
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 940,
        "y": 500,
        "wires": [
            [
                "8a2ab246ddd8bb42"
            ]
        ]
    },
    {
        "id": "8a2ab246ddd8bb42",
        "type": "file",
        "z": "2d177c76a333981b",
        "name": "",
        "filename": "/tmp/chart_raw.log",
        "filenameType": "str",
        "appendNewline": true,
        "createDir": false,
        "overwriteFile": "true",
        "x": 1140,
        "y": 500,
        "wires": [
            []
        ]
    },
    {
        "id": "3cba6ed55fa3c2fb",
        "type": "mqtt-broker",
        "name": "MQTT server",
        "broker": "localhost",
        "port": "1883",
        "clientid": "",
        "autoConnect": true,
        "usetls": false,
        "protocolVersion": "4",
        "keepalive": "60",
        "cleansession": true,
        "autoUnsubscribe": true,
        "birthTopic": "",
        "birthQos": "0",
        "birthRetain": "false",
        "birthPayload": "",
        "birthMsg": {},
        "closeTopic": "",
        "closeQos": "0",
        "closeRetain": "false",
        "closePayload": "",
        "closeMsg": {},
        "willTopic": "",
        "willQos": "0",
        "willRetain": "false",
        "willPayload": "",
        "willMsg": {},
        "userProps": "",
        "sessionExpiry": ""
    },
    {
        "id": "bb43d51ed33cfd60",
        "type": "ui_group",
        "name": "Default",
        "tab": "43dd28532aa9ae86",
        "order": 1,
        "disp": true,
        "width": "6",
        "collapse": false,
        "className": ""
    },
    {
        "id": "43dd28532aa9ae86",
        "type": "ui_tab",
        "name": "Dataloggers",
        "icon": "dashboard",
        "disabled": false,
        "hidden": false
    }
]

Here is what the dashboard looks like with some data coming in via MQTT. Because a new MQTT message comes every second, the entire chart is refreshing every second (instead of a second's worth of new data being appended to the end).

chart
chart506Ă—522 21.4 KB

Some part of me wonders whether I should instead use Node Red to send the data from MQTT to a different system like InfluxDB and then use something like Grafana to plot/save it, but I'm trying to avoid using too many different things talking to each other if possible.

2

You have a syntax error in your function node, CH1 has [ ] around the push object. That is why you only see 3 lines.

If you wish to append the data with dashboard 1 you will have to save the chart data to context then add the new incoming data, then send the new array object to the chart.

Or split the incoming message and send the individual payload, topics and timestamps to the chart

e.g

[{"id":"7625c1316a27c191","type":"inject","z":"2d177c76a333981b","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"Ch1\":[[3716421,3716479,3716545,3716613,3716680,3716747,3716815,3716883,3716950,3717018,3717085,3717152,3717220,3717288,3717356],[0.000666,0.000662,0.000658,0.000659,0.000663,0.000662,0.000658,0.00066,0.00066,0.000657,0.000662,0.000657,0.000654,0.00065,0.000654]],\"Ch2\":[[3716426,3716494,3716561,3716629,3716697,3716764,3716832,3716899,3716967,3717034,3717102,3717169,3717237,3717304,3717372],[-0.000449,-0.000452,-0.000457,-0.000454,-0.000458,-0.000461,-0.000458,-0.000462,-0.000466,-0.000463,-0.000461,-0.000463,-0.000464,-0.000461,-0.000459]],\"Ch3\":[[3716443,3716511,3716578,3716646,3716713,3716781,3716848,3716916,3716984,3717051,3717119,3717186,3717254,3717321,3717389],[0.000359,0.000359,0.000358,0.000358,0.000358,0.000358,0.000356,0.000358,0.000359,0.000359,0.000358,0.000359,0.000361,0.000359,0.000362]],\"Ch4\":[[3716460,3716528,3716595,3716663,3716730,3716798,3716865,3716933,3717000,3717068,3717135,3717203,3717270,3717338,3717406],[0.002398,0.002397,0.002397,0.002395,0.002396,0.002395,0.002392,0.00239,0.00239,0.002387,0.002387,0.002385,0.002384,0.002384,0.002381]],\"ID\":\"00E5002C\"}","payloadType":"json","x":410,"y":180,"wires":[["369f80752f62d5ab","69c772af70b22a2b"]]},{"id":"369f80752f62d5ab","type":"debug","z":"2d177c76a333981b","name":"debug 1","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":540,"y":300,"wires":[]},{"id":"69c772af70b22a2b","type":"change","z":"2d177c76a333981b","name":"","rules":[{"t":"set","p":"payload.ID","pt":"msg","to":"ID","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":590,"y":180,"wires":[["f688df8f9ae483a3"]]},{"id":"f688df8f9ae483a3","type":"split","z":"2d177c76a333981b","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"topic","property":"payload","x":770,"y":180,"wires":[["fab3786c7e3e973b"]]},{"id":"fab3786c7e3e973b","type":"change","z":"2d177c76a333981b","name":"","rules":[{"t":"set","p":"timestamp","pt":"msg","to":"payload[0]","tot":"msg","dc":true},{"t":"set","p":"payload","pt":"msg","to":"payload[1]","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":920,"y":180,"wires":[["5ceee031047e5b45"]]},{"id":"5ceee031047e5b45","type":"split","z":"2d177c76a333981b","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","property":"payload","x":610,"y":240,"wires":[["2a11a690dda0e93b"]]},{"id":"2a11a690dda0e93b","type":"change","z":"2d177c76a333981b","name":"","rules":[{"t":"set","p":"timestamp","pt":"msg","to":"timestamp[msg.parts.index]","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":770,"y":240,"wires":[["09a8a59620e214af","346d14cb602666a0"]]},{"id":"09a8a59620e214af","type":"debug","z":"2d177c76a333981b","name":"debug 2","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":840,"y":300,"wires":[]},{"id":"346d14cb602666a0","type":"ui_chart","z":"2d177c76a333981b","name":"","group":"bb43d51ed33cfd60","order":0,"width":"6","height":"6","label":"Datalogger 1","chartType":"line","legend":"true","xformat":"HH:mm:ss","interpolate":"linear","nodata":"","dot":true,"ymin":"","ymax":"","removeOlder":"110000","removeOlderPoints":"1000","removeOlderUnit":"604800","cutout":0,"useOneColor":false,"useUTC":false,"colors":["#1f77b4","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"outputs":1,"useDifferentColor":false,"className":"","x":1010,"y":240,"wires":[["7cd9833233fb8a86"]]},{"id":"476cd736b2739655","type":"ui_button","z":"2d177c76a333981b","name":"Clear","group":"bb43d51ed33cfd60","order":1,"width":"2","height":"1","passthru":false,"label":"Clear chart","tooltip":"","color":"","bgcolor":"","className":"","icon":"","payload":"[]","payloadType":"json","topic":"","topicType":"str","x":670,"y":140,"wires":[["346d14cb602666a0"]]},{"id":"7cd9833233fb8a86","type":"trigger","z":"2d177c76a333981b","name":"","op1":"","op2":"","op1type":"nul","op2type":"payl","duration":"250","extend":true,"overrideDelay":false,"units":"ms","reset":"","bytopic":"all","topic":"topic","outputs":1,"x":1080,"y":280,"wires":[["c493d2ea78045827","72d1d41e3df2ae4e"]]},{"id":"c493d2ea78045827","type":"debug","z":"2d177c76a333981b","name":"debug 3","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1140,"y":320,"wires":[]},{"id":"72d1d41e3df2ae4e","type":"function","z":"2d177c76a333981b","name":"","func":"if (msg.topic === \"save\") {\n    msg.payload = context.get(\"last\");\n    return msg;\n}\nelse {\n    context.set(\"last\", msg.payload);\n}\nreturn null;","outputs":1,"timeout":"","noerr":0,"initialize":"","finalize":"","libs":[],"x":940,"y":500,"wires":[["8a2ab246ddd8bb42"]]},{"id":"1ad58c6121419e17","type":"ui_button","z":"2d177c76a333981b","name":"","group":"bb43d51ed33cfd60","order":3,"width":"2","height":"1","passthru":false,"label":"Save Charts","tooltip":"","color":"","bgcolor":"","className":"","icon":"","payload":"","payloadType":"str","topic":"save","topicType":"str","x":690,"y":500,"wires":[["72d1d41e3df2ae4e"]]},{"id":"8a2ab246ddd8bb42","type":"file","z":"2d177c76a333981b","name":"","filename":"/tmp/chart_raw.log","filenameType":"str","appendNewline":true,"createDir":false,"overwriteFile":"true","x":1140,"y":500,"wires":[[]]},{"id":"bb43d51ed33cfd60","type":"ui_group","name":"Default","tab":"43dd28532aa9ae86","order":1,"disp":true,"width":"6","collapse":false,"className":""},{"id":"43dd28532aa9ae86","type":"ui_tab","name":"Dataloggers","icon":"dashboard","disabled":false,"hidden":false}]

But maybe you should start using dashboard 2 https://dashboard.flowfuse.com/

You may well be better off using Influx and Grafana

Also you are not using the correct syntax for context store, this may bite you later on Writing Functions : Node-RED

1 Like
3

Thanks, this is very helpful! Seems like using v2 is going to work much better (will need to adjust the function to put the data in a payload appropriate for a v2 chart, but at least it has append). I'll work on the save feature after (that was imported from a different flow for a different purpose)

4

I've got it all working nicely with Dashboard V2.0, but now wondering about how to best save the data. It looks like the chart output is just the input message, not the entirety of the chart (when operating in append mode). Is there any way to change this such that the output is the entire chart? If not, I'm thinking I'll need to append each data "chunk" to a variable in context, and then save that when the save button is pressed? Basically what I want is that when the Save button is pushed, the entire chart contents is written to a file (overwriting the file, and if possible with user adjustable file name)

5

No matter which kind, the dashboard is for data representation and interface for user communication.

It is wrong to use it for something else.
Specially for data storage and data formatting.

Such kind of things should always be done at the server side, for bigger datasets the database is one and only proper tool.

The amount of the data feed to dashboard should be kept as minimal as possible.

Ignorance of any of those rules leads to bad performance., many cases to browser crash but definitely to frustration.

6

Do you mean that dashboard 1 got it wrong in providing an output that one could write to context and pick up on restart to restore the chart?

7

Um, these answers are not really helpful or answering the question. If what you meant to say was "Hey, perhaps a better way to do this would be to ABC" (where, I dunno, ABC is having the function send each chunk of data to a database or something, and then the chart pull from said database for a visual representation, providing details about what you are suggesting), then that could have been helpful. But just saying "you are doing it wrong" without knowing the context of what is being done isn't really all that helpful.

How many points on a chart is too many? Where is the documentation for that? What is a
"bigger" dataset? 10,000 entries? 10,000,000 entries? 1GB of data? How many data points do I anticipate? (probably a fair question to ask before trashing the approach...)

8

I do. I see it as an easy way to achieve low code/no code. But that all comes with the cost and side effects.

Seems to me you do know. This is the right way. But definitely not shortest or easiest.

There's no answer to question about how much data is too much. It all depends on many many factors and can be settled per case only if all factors are known.

9

The chart in Dashboard 1 was designed to be updated in two ways. The first was with live data - just add the points to the existing chart. and the second was to replace the whole chart - by sending an array of all the data. There is no halfway house to add a group of data.

So if it is like live data then split your chunks into individual points and add them (with or without timestamp - depending if they arrive with timestamp or not). In general this is a much better approach than trying to re-write the entire chart every time data appears (I think you said once per second).

As regards "too many points" - the general thinking is that by default (in dashboard 1) the group is 6 units of 50 pixels wide (approx) = 300 pixels wide per chart - so having many more points than that will just be unproductive. (Yes I know the rendering may smear the points somehow so you can see some extra data but there is no point in sending 3000 points to fit 300 pixels) - so you need to size your data window appropriately for what you are trying to display.