Storing Chart Data from a D2 Dashboard

[DarkZone]

Hey guys, I'm working on a few Dashboards and have found it pretty annoying to lose all my charted points when im restarting my Docker instance.

I tried to read up on it using this flowfuse article but noticed it is for Dashboard-1 and not 2. If i attach a Debug Node to the Chart Node from D2 i just get the message that is passed into the Chart Node, even if my Chart Node is set to "Append" and not "Replace".

Looking at the article it seemed so easy to persist the data, but I couldnt find any way to do it with the Dashboard2, is there a best practice way to do it?

Thanks for the help.

[joepavitt]

Hi @DarkZone - it has come up as a request before, and something we're considering bringing back.

The logic behind it with Dashboard 2.0 is that you can just store a list of your messages, pass them back in, and it'll render the data accordingly.

[DarkZone]

Hello Joe thanks for the answer,

I'm afraid that's not working properly, propably my mistake tho. I made a function that now trims down my mqtt messages to a 3 key object. it is structured like so : msg.payload ={"timestamp":1725967766338,"value":25.5,"sourcePath":"Innen-Links"}

I append those messages to a file in seperate lines. I then read the file line by line, with a message being sent for every line. the messages payload, e.g. : "
{"timestamp":1725967766338,"value":25.5,"sourcePath":"Inside-Left"}" is being converted to a json object and then passed into the chart. but for some reason i have duplicates?

Here is the complete flow:

flows.json (8.3 KB)

[smcgann99]

@DarkZone

Chart Node is set to "Append" and not "Replace".

Perhaps this is your problem ?

[joepavitt]

I second this. Im guessing you're parsing multiple times, and the data just continues to append. We don't check for duplicates before adding to the chart, as it's perfectly valid for a chart to have two points of data with the same values.

[DarkZone]

Perhaps there is a misunderstanding, my current flow looks like this

using the replace instead of append option really messes with the graph since each value retrieved over the mqtt subscriber is replacing the previous value.

The desired functionality is: Chart out the retrieved JSON data from the mqtt-in Nodes. If the instance were to restart it should fetch each value of the temp-data.txt and place it in the chart and continue with the values that are once again coming in through the mqtt-in Nodes.

That should only be able with the append option checked, right?

The function node just trims down on the JSON Data from the mqtt nodes:

// Extract the relevant data from the MQTT message
var msg = {
    timestamp: msg.payload.data[0].values[0].timestamp,
    value: msg.payload.data[0].values[0].value,
    sourcePath: msg.payload.data[0].sourcePath
};

if (msg.sourcePath.indexOf('Innen-Links') >= 0) {
    msg.sourcePath = 'Innen-Links';
}
if (msg.sourcePath.indexOf('Innen-Rechts') >= 0) {
    msg.sourcePath = 'Innen-Rechts';
}
if (msg.sourcePath.indexOf('Aussen-Links') >= 0) {
    msg.sourcePath = 'Aussen-Links';
}
if (msg.sourcePath.indexOf('Aussen-Rechts') >= 0) {
    msg.sourcePath = 'Aussen-Rechts';
}
// Set the extracted data to the new message payload
return { payload: msg };

[E1cid]

You can store the incoming data to a database or depending how many points your chart is showing a context variable. You can then restore your chart using the saved data.

If you require more help i suggest you show an example of your incoming data (in text form) and an export of your flow. You can use the debug sidebar too copy data.

There’s a great page in the docs (Working with messages : Node-RED) that will explain how to use the debug panel to find the right path/value for any data item.

Pay particular attention to the part about the buttons that appear under your mouse pointer when you over hover a debug message property in the sidebar.

How to import/export a flow

In order to make code readable and usable it is necessary to surround your code with three backticks (also known as a left quote or backquote ```)

``` 
   code goes here 
```

You can edit and correct your post by clicking the pencil icon.

See this post for more details - How to share code or flow json

[joepavitt]

Okay, so you're definitely taking the correct approach with "Append", however, it does seem something in your logic is causing the same data to be appended multiple times.

[DarkZone]

Hello @E1cid,

the mqtt-messages currently look like this before getting transformed by the function node: "{"version":"1.0","data":[{"processDataUnit":"°C","tid":"31e4d5ae-540a-4c46-b2b0-c6146022b153","psid":"Temperature","devicePath":"V5-KO-INO-MaschinenNR/10.1.50.4:80/LDH292/X01/Temperature","sourcePath":"Test / Deutschland /<redacted>/ V5-KO-INO-MaschinenNR / V5-KO-INO-MaschinenNR / Aussen-Rechts / Temperature","values":[{"timestamp":1726476465180,"value":23.6}]}]}.

After being processed by the function node they are shortened to this: {"timestamp":1726476735774,"value":23.6,"sourcePath":"Aussen-Rechts"}

I provided the flow earlier in my initial post, but I will attach it to this message again since i changed it a bit.
flows.json (9.5 KB)

[DarkZone]

Hello @joepavitt ,

I added the flow once again in my previous comment. I currently still cant get behind why it wouldnt work in the current config. Maybe you can gather something from it.

Thanks for taking a look.

[E1cid]

Here is an example of how I would save data to context and recover when server restarted.
You will need to set your context store to persistent Working with context : Node-RED

I have limited the stored data to 1000 readings,

I have edited all nodes so look at them all.
Ì have included test data, this is how you should share data, attach it to your flow as it is helpful for others to test, as they do not have access to your mqtt nodes.

Attaching files is not best practice when sharing exported flows, unless the file is to large. You should attach flow.jsons as i described above.

e.g

[{"id":"f4d9cb6bf96478c7","type":"mqtt in","z":"d1395164b4eec73e","name":"all sensors","topic":"moneo-server/Test/Deutschland/V5-KO-INO-MaschinenNR/+/Temperature","qos":"2","datatype":"auto-detect","broker":"4ce6fef99d553369","nl":false,"rap":true,"rh":0,"inputs":0,"x":223.00001525878906,"y":5901.000091075897,"wires":[["a9337d9956e1469b"]]},{"id":"a9337d9956e1469b","type":"function","z":"d1395164b4eec73e","name":"Convert message","func":"//set allowed snsors\nconst allowed_sensors = [\n    'Innen-Links', \n    'Innen-Rechts', \n    'Aussen-Links', \n    'Aussen-Rechts'\n]\n// extract sourcePath\nconst sourcePath = msg.payload.data[0].sourcePath.split(\"/\")[5].trim();\n\n// continue if allowed sensor\nif (allowed_sensors.includes(sourcePath)) {\n// Extract the relevant data from the MQTT message\n    msg.payload = {\n        timestamp: msg.payload.data[0].values[0].timestamp,\n        value: msg.payload.data[0].values[0].value,\n        sourcePath: sourcePath\n    };\n// save data \n    let chart_data = flow.get(\"chart_data\") ?? [];\n    chart_data.push(msg.payload);\n    chart_data = chart_data.slice(-1000);\n    flow.set(\"chart_data\", chart_data)\n    \n    return msg\n}","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":493.00001525878906,"y":5901.000091075897,"wires":[["05e73d8550f72e07","67c7d7a139f737ac"]]},{"id":"05e73d8550f72e07","type":"ui-chart","z":"d1395164b4eec73e","group":"fedf084a329203c8","name":"","label":"Temperatur Schleifen","order":1,"chartType":"line","category":"sourcePath","categoryType":"property","xAxisLabel":"Datum","xAxisProperty":"timestamp","xAxisPropertyType":"property","xAxisType":"time","xAxisFormat":"","xAxisFormatType":"HH:mm:ss","yAxisLabel":"","yAxisProperty":"value","ymin":"","ymax":"","action":"append","stackSeries":false,"pointShape":"false","pointRadius":4,"showLegend":true,"removeOlder":"3","removeOlderUnit":"86400","removeOlderPoints":"","colors":["#1f77b4","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"textColor":["#666666"],"textColorDefault":true,"gridColor":["#e5e5e5"],"gridColorDefault":true,"width":6,"height":"4","className":"","x":803.0000152587891,"y":5901.000091075897,"wires":[[]]},{"id":"67c7d7a139f737ac","type":"debug","z":"d1395164b4eec73e","name":"debug 2570","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":710,"y":5780,"wires":[]},{"id":"798346bd74f34d55","type":"function","z":"d1395164b4eec73e","name":"function 1","func":"msg.payload =flow.get(\"chart_data\") ?? []\nmsg.action = \"replace\"\nreturn msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":503.00001525878906,"y":6101.000091075897,"wires":[["05e73d8550f72e07"]]},{"id":"4ba63a8f241c68cb","type":"inject","z":"d1395164b4eec73e","name":"reset chart","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"[]","payloadType":"json","x":423.00001525878906,"y":5961.000091075897,"wires":[["05e73d8550f72e07"]]},{"id":"8744fb9558b3fc64","type":"inject","z":"d1395164b4eec73e","name":"","props":[],"repeat":"","crontab":"","once":true,"onceDelay":0.1,"topic":"","x":333.00001525878906,"y":6101.000091075897,"wires":[["798346bd74f34d55"]]},{"id":"57775e96caecdebc","type":"inject","z":"d1395164b4eec73e","name":"test data Innen-Rechts","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"moneo-server/Test/Deutschland/V5-KO-INO-MaschinenNR/Innen-Rechts/Temperature","payload":"{\t    \"version\":\"1.0\",\t    \"data\":[\t        {\t            \"processDataUnit\":\"°C\",\t            \"tid\":\"31e4d5ae-540a-4c46-b2b0-c6146022b153\",\t            \"psid\":\"Temperature\",\t            \"devicePath\":\"V5-KO-INO-MaschinenNR/10.1.50.4:80/LDH292/X01/Temperature\",\t            \"sourcePath\":\"Test / Deutschland /<redacted>/ V5-KO-INO-MaschinenNR / V5-KO-INO-MaschinenNR / Innen-Rechts / Temperature\",\t            \"values\":[\t                {\t                    \"timestamp\":$millis(),\t                    \"value\": $round($random()*10+10, 1)\t                }\t            ]\t        }\t    ]\t}","payloadType":"jsonata","x":300,"y":5800,"wires":[[]]},{"id":"4ce6fef99d553369","type":"mqtt-broker","name":"emqx","broker":"emqx","port":"1883","clientid":"","autoConnect":true,"usetls":false,"protocolVersion":"4","keepalive":"60","cleansession":true,"autoUnsubscribe":true,"birthTopic":"","birthQos":"0","birthRetain":"false","birthPayload":"","birthMsg":{},"closeTopic":"","closeQos":"0","closeRetain":"false","closePayload":"","closeMsg":{},"willTopic":"","willQos":"0","willRetain":"false","willPayload":"","willMsg":{},"userProps":"","sessionExpiry":"","info":"Benutzt den namen des Docker-Containers direkt da sie im \r\nselben Netzwerk sind und sich über DNS finden können"},{"id":"fedf084a329203c8","type":"ui-group","name":"Temperatur Schleifen","page":"2693a6e13851f0fd","width":"6","height":"1","order":1,"showTitle":true,"className":"","visible":"true","disabled":"false"},{"id":"2693a6e13851f0fd","type":"ui-page","name":"IFM Dashboard","ui":"1805777f90e92057","path":"/IFM","icon":"home","layout":"grid","theme":"576394c7e38e257b","order":3,"className":"","visible":true,"disabled":false},{"id":"1805777f90e92057","type":"ui-base","name":"dashboard","path":"/dashboard","includeClientData":true,"acceptsClientConfig":["ui-notification","ui-control"],"showPathInSidebar":false,"showPageTitle":true,"titleBarStyle":"default"},{"id":"576394c7e38e257b","type":"ui-theme","name":"Default Theme","colors":{"surface":"#ffffff","primary":"#0094CE","bgPage":"#eeeeee","groupBg":"#ffffff","groupOutline":"#cccccc"},"sizes":{"pagePadding":"12px","groupGap":"12px","groupBorderRadius":"4px","widgetGap":"12px"}}]

[edit] update function to work with your supplied data format

[DarkZone]

Thank you for the input, I was too obsessed with using the msg. option when trying to assign a series. using the key as the series identifier makes it much simpler, especially when the message is transformed by functions and injection nodes.

I will keep your tips in mind regarding the etiquette, I didn't give it enough thought. In hindsight it is pretty obvious that uploading the whole flow as a file doesn't make much sense.

Kinde regards

[DarkZone]

@joepavitt @smcgann99 @E1cid

I have found out why I have duplicate values in the chart. Everytime I deploy my injection-node that is supposed to load the data on a restart, injects the data and appends it to the chart. I guess I will put another injection-node and use that to clear the chart.

I didn't know it injects on every deploy.

Kind regards

[E1cid]

If you look at the function that retrieves the context data to restore the chart, you will see it uses msg.action = "replace". This tells the chart to replace the data rather than append, then there is no need to sent an empty array to reset the chart You can simply add a msg.replace to the inject node.

[Steve-Mcl]

@DarkZone

I didn't know it injects on every deploy

You are probably doing a full deploy. The deploy button has a drop down next to it. You can change it to do different types of deployment.

Essentially a full deploy destroys and recreates every single node. That means your inject will trigger.

[Godfrey]

Thank you