# Problem with custom UI node and replaying data
**URL:** https://discourse.nodered.org/t/problem-with-custom-ui-node-and-replaying-data/49478
**Category:** Dashboard
**Tags:** node-red-dashboard
**Created:** [8 August 2021 10:31 UTC](https://discourse.nodered.org/t/problem-with-custom-ui-node-and-replaying-data/49478 "2021-08-08T10:31:52Z")
**Posts on this page:** 18
**Page:** 1
### Author:  [@Christian-Me](https://discourse.nodered.org/u/Christian-Me)
#### Post date: [8 August 2021 10:31 UTC](https://discourse.nodered.org/t/problem-with-custom-ui-node-and-replaying-data/49478/1 "2021-08-08T10:31:52Z")
Hello,
I have a problem with replaying persistent data creating a custom chart ui-widget. After hours of debugging perhaps someone can help:
- every message creates a new datapoint on the dashboard (works as expected)
- I use the convert function to create the new datapoint `conversion.newPoint`
- I push the new point to `conversion.updatedValues`
- I set the `conversion.update`flag to `true`
_shorted Version_
```auto
convert: function(value,fullDataset,msg,step) {
var conversion = {
updatedValues: [],
newPoint: {},
}
if (fullDataset) conversion.updatedValues = fullDataset;
conversion.newPoint.state = RED.util.getMessageProperty(msg, config.stateField || "payload");
conversion.newPoint.topic = msg.topic;
conversion.updatedValues.push(conversion.newPoint);
conversion.update = true;
return conversion;
},
```
- then `beforeEmit` is called twice, once with the `fullDataset` I return and it is stored into replay messages and next with the `newPoint`. (I later found out I perhaps could have done with the `beforeEmit` function only)
```auto
beforeEmit: function(msg, fullDataset) {
if (Array.isArray(fullDataset)) { return fullDataset };
var newMsg = fullDataset;
if (msg) {
if (msg.socketid) newMsg.socketid = msg.socketid;
}
return { msg: newMsg };
},
```
- using the debugger I can observe that `replayMessages[nodeId]` is holding the correct array of datapoints as expected
My problem is that the `replayMessages` array is never sent to the dashboard. Not on tab changes and not after page connects ☹ Replaying the last message works but getting the historic data does not work.
Perhaps someone can shine some light on the correct mechanics of preparing, sending and replaying messages from the backend to the ui. Any working example _(myLittleNode does not capture that topic and the original chart node seams not using the official API)_
---
### Author:  [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)
#### Post date: [8 August 2021 18:38 UTC](https://discourse.nodered.org/t/problem-with-custom-ui-node-and-replaying-data/49478/2 "2021-08-08T18:38:54Z")
Hi @Christian-Me,
It is now some ago that I created a UI node, but indeed the msg replay was a battle every time...
I don't remember all the `addWidget` parameters anymore, but I assume you have been experimenting with those...
Perhaps you can share your `addWidget` parameters here, so everybody knows what you are using...
Bart
---
### Author:  [@Christian-Me](https://discourse.nodered.org/u/Christian-Me)
#### Post date: [8 August 2021 21:15 UTC](https://discourse.nodered.org/t/problem-with-custom-ui-node-and-replaying-data/49478/3 "2021-08-08T21:15:19Z")
Hi Bart,
thank you offering your help 🙂 All based on myLittleUiNode (... working fine with my iro-color-picker, of which perhaps some fragments are still in the code) ...
```auto
var done = ui.addWidget({ // *REQUIRED* !!DO NOT EDIT!!
type: 'chart',
label: config.label,
tooltip: config.tooltip,
node: node, // *REQUIRED* !!DO NOT EDIT!!
order: config.order, // *REQUIRED* !!DO NOT EDIT!!
group: config.group, // *REQUIRED* !!DO NOT EDIT!!
width: config.width, // *REQUIRED* !!DO NOT EDIT!!
height: config.height, // *REQUIRED* !!DO NOT EDIT!!
format: html, // *REQUIRED* !!DO NOT EDIT!!
templateScope: "local", // *REQUIRED* !!DO NOT EDIT!!
emitOnlyNewValues: false, // *REQUIRED* Edit this if you would like your node to only emit new values.
forwardInputMessages: false, // *REQUIRED* Edit this if you would like your node to forward the input message to it's ouput.
storeFrontEndInputAsState: false, // *REQUIRED* If the widget accepts user input - should it update the backend stored state ?
```
`persistantFrontEndValue` is missing but is `true` by default.
here I could trace it down into `ui.js` that the replay messages are stored:
```auto
if (opt.persistantFrontEndValue === true) {
replayMessages[opt.node.id] = toStore;
}
```
attached the work in progress [ui-uplot-charts.js.txt](https://discourse.nodered.org/uploads/short-url/yrKtvZYW4RYSBCKdRw2ppwsKwG0.txt) (15.8 KB)
---
### Author:  [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)
#### Post date: [8 August 2021 21:47 UTC](https://discourse.nodered.org/t/problem-with-custom-ui-node-and-replaying-data/49478/4 "2021-08-08T21:47:54Z")
It is already some time ago that I have been using those parameters...
Note that a _ **tab switch** _ works differently, because then the message will be replayed on the client side (i.e. in the dashboard)! See [here](https://discourse.nodered.org/t/msg-replayed-in-ui-node-after-a-tab-switch/25032/4) for more info. While in the other cases the replaying is triggered by the server side of your node ... But that can perhaps explain why replaying the last message only adds a single point???
From another [discussion](https://discourse.nodered.org/t/refresh-ui-node/13921/24) I think that the message is stored in replaymessages (for the current node), as soon as the message arrives in your node. Do I understand correctly that you afterwards replace it by a full history of points (since the last msg only contained a single point?)??
My time is up for today ...
---
### Author:  [@Christian-Me](https://discourse.nodered.org/u/Christian-Me)
#### Post date: [9 August 2021 00:18 UTC](https://discourse.nodered.org/t/problem-with-custom-ui-node-and-replaying-data/49478/5 "2021-08-09T00:18:49Z")
Think I found the problem and the solution! (nothing better than a Gin and Tonic and a relaxed debugging night session when the house is quiet 😉
the property `updatedValues` must be an object and must contain a `msg` property ...
Something similar to this does the job
```auto
convert: function(value,fullDataset,msg,step) {
var conversion = {
updatedValues: {msg:{payload:[]}},
newPoint: {},
}
if (fullDataset) conversion.updatedValues = fullDataset;
conversion.newPoint.state = RED.util.getMessageProperty(msg, config.stateField || "payload");
conversion.newPoint.topic = msg.topic;
conversion.updatedValues.msg.payload.push({conversion.newPoint.topic:conversion.newPoint.state});
conversion.update = true;
return conversion;
},
```
as so often we (including myself) are particular bad in naming things. Instead `updatedValues` perhaps something like `updatedMessages` or perhaps better `replayMessage` could have given me the right clue and saved some time digging into `socket.js` 😉
---
### Author:  [@Christian-Me](https://discourse.nodered.org/u/Christian-Me)
#### Post date: [9 August 2021 11:03 UTC](https://discourse.nodered.org/t/problem-with-custom-ui-node-and-replaying-data/49478/6 "2021-08-09T11:03:45Z")
Ok I solved the reconnect problem = I get the accumulated dataset from the backend ... but the tab changed problem is still an issue.
After a tab change I get the last message replayed ... fine, but the accumulated data (I collected while widget was visible) in `$scope._data` is gone ☹ I any way this is ok as over time the tab was not visible new datapoints could have arrived, so I need the up to date data from the backend.
**Question is :** How can I let the client ask for the `updatedValues.msg` after a tab change instead getting only the last message (`newPoint`).
I could send always the complete data set every time but this would be a waste of bandwidth (one intention to use uPlot is to be capable of huge datasets and fast updates).
---
### Author:  [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)
#### Post date: [9 August 2021 13:38 UTC](https://discourse.nodered.org/t/problem-with-custom-ui-node-and-replaying-data/49478/7 "2021-08-09T13:38:23Z")
> [@Christian-Me](#):
>
> How can I let the client ask for the `updatedValues.msg` after a tab change instead getting only the last message ( `newPoint` ).
As you can see [here](https://discourse.nodered.org/t/msg-replayed-in-ui-node-after-a-tab-switch/25032/5), we used a trick to see whether a message is being replayed on the client side (e.g. by a tab switch): add a field to the input msg (i.e. on the client side copy of the input msg). When that field is already available when the watch is triggered, then you know that it is a client-side replayed msg (which has previously been handled already by this watch...). In that case you could get your complete dataset from the server.
But of course it is only a trick...
---
### Author:  [@Christian-Me](https://discourse.nodered.org/u/Christian-Me)
#### Post date: [9 August 2021 14:12 UTC](https://discourse.nodered.org/t/problem-with-custom-ui-node-and-replaying-data/49478/8 "2021-08-09T14:12:49Z")
Hmmm perhaps I need a G&T 🤪
I'm aware of "rebouncing" messages .... Your trick is nice (and better than my solution in iro-color), but I think this is not my problem as the widget does not emit messages (display only).
On Tab changes I only get the last datapoint ... but I need in this case the complete dataset to (re-)initialize the chart.
On connect get exactly the message I need and easy to identify as `msg.fullDataset`
Tab changes after a connect (with no later updates) are fine as the last message has the full dataset.
But how I can get the backend to send me the full dataset (instead the last datapoint)?
---
### Author:  [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)
#### Post date: [9 August 2021 16:14 UTC](https://discourse.nodered.org/t/problem-with-custom-ui-node-and-replaying-data/49478/9 "2021-08-09T16:14:55Z")
> [@Christian-Me](#):
>
> as my node does not emit any messages.
Now I'm lost. How can you get messages (and replayed messages) in the dashboard side, when your node does not emit messages, i.e. push messages from server-side to client-side (after calling your beforeEmit function)?
---
### Author:  [@Christian-Me](https://discourse.nodered.org/u/Christian-Me)
#### Post date: [9 August 2021 17:34 UTC](https://discourse.nodered.org/t/problem-with-custom-ui-node-and-replaying-data/49478/10 "2021-08-09T17:34:09Z")
Sorry, I was unclear ... the widget does not emit messages (òutputs=0)
Internally the node (backend) emits messages to the client (frontend).
---
### Author:  [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)
#### Post date: [9 August 2021 19:24 UTC](https://discourse.nodered.org/t/problem-with-custom-ui-node-and-replaying-data/49478/11 "2021-08-09T19:24:41Z")
> [@Christian-Me](#):
>
> the widget does not emit messages (òutputs=0)
Yes in the beginning I was struggling with the names:
- The ui node _ **emits** _ (=pushes) messages via socketio from server to client. In the server you can update the messages in ´beforeEmit´, and in the client you can ´watch´ them.
- The ui node _ **sends** _ messages on the server side to its output, and you can update them in ´beforeSend´.
> [@Christian-Me](#):
>
> But how I can get the backend to send me the full dataset (instead the last datapoint)
I "think" you cannot push them from the server to the client, since the server-side of your node is not aware of this happening. But I might be mistaken!! Hopfully somebody else can confirm or deny this...
That would lean that the client needs to get the data from the server. For example when the watch sees a replayed message, it:
- calls an endpoint of your node
- sends a msg to the server (e.g. topic="getfulldata"), which you handle in beforeSend by pushing the full data msg to the client
Not sure if my proposals make sense, because I haven't tried it!
---
### Author:  [@Christian-Me](https://discourse.nodered.org/u/Christian-Me)
#### Post date: [9 August 2021 20:09 UTC](https://discourse.nodered.org/t/problem-with-custom-ui-node-and-replaying-data/49478/12 "2021-08-09T20:09:41Z")
That sounds like a pretty good concept to me! Thank you! Will walk the 🐕 grab me a 🍸and give it a try. Already see some challenges
- distinguish a tab change from a normal reconnect
- then send the complete dataset to a specific socketid only
But I also remember something similar I found in the original chart node (think it was a onTabChange event listener I could not get working in the first try as the api is different)
So some options to play with 👍
---
### Author:  [@Christian-Me](https://discourse.nodered.org/u/Christian-Me)
#### Post date: [9 August 2021 23:04 UTC](https://discourse.nodered.org/t/problem-with-custom-ui-node-and-replaying-data/49478/13 "2021-08-09T23:04:04Z")
After a while of trying to emit a message directly (failing to get access to `emitSocket()` I found out that is is possible to send a message to myself via `node.receive()`
For anybody interested or anybody saying I'm doing it all wrong here are some code fragments:
1. the $watch code:
```auto
$scope.$watch('msg', function(msg) {
if (!msg) { return; } // Ignore undefined msg
if ($scope._data[0]===undefined) {
console.log('uPlot first Message > asking for replay');
$scope.send({payload:"R"});
return;
}
if (msg.hasOwnProperty('fullDataset')) {
console.log('uPlot fullDataset received:',msg);
// do something with the full dataset
});
} else if (isNewDataPoint(msg)){
console.log('uPlot dataPoint received:',msg);
// do something with the datapoint
}
});
```
1. in `beforeSend` trigger let's trigger a message to myself
```auto
beforeSend: function (msg, orig) {
if (orig.msg.payload === 'R') {
node.receive(orig.msg);
return;
}
if (orig) {
var newMsg = {};
// do something here if your node emits data
}
},
```
1. in `convert` return the full dataset if a replay is requested
```auto
convert: function(value,fullDataset,msg,step) {
if (msg.payload==='R') {
return fullDataset;
}
var conversion = {
updatedValues: {
msg:{
fullDataset : []
}},
newPoint: {},
}
// do your converting business
return conversion;
},
```
1. in `beforeEmit` return the full dataset again (if present and well formed)
```auto
beforeEmit: function(msg, fullDataset) {
if (fullDataset && fullDataset.hasOwnProperty('msg') && fullDataset.msg.hasOwnProperty('fullDataset')) {
return fullDataset
};
// prepare your payload ready for launch
return { msg: newMsg };
},
```
and then you should get something like this if you switch the tab or refresh your page:
later messages simply add a new datapoint:
Let's see if this solution works in a longer development and debugging period
and all of this will later perhaps look like this (just a proof of concept and my own capabilities)
---
### Author:  [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)
#### Post date: [13 August 2021 06:02 UTC](https://discourse.nodered.org/t/problem-with-custom-ui-node-and-replaying-data/49478/14 "2021-08-13T06:02:13Z")
Hi @Christian-Me,
Sorry for the lare reply!
Thanks for sharing the code sippets, which summarize nicely your mechanism! But can you also share a link to your repo (if available), because some things are not clear to me:
- When I see `function(msg, fullDataset)` it looks to me you always send a full dataset to the frontend. But that is not the case I assume?
- Where the convert function is being called.
- When $scope.\_data[0] is being filled
---
### Author:  [@Christian-Me](https://discourse.nodered.org/u/Christian-Me)
#### Post date: [14 August 2021 10:50 UTC](https://discourse.nodered.org/t/problem-with-custom-ui-node-and-replaying-data/49478/15 "2021-08-14T10:50:50Z")
Hi @BartButenaers
> [@BartButenaers](#):
>
> But can you also share a link to your repo (if available)
Will do as soon I cleaned up some of the mess and get the basic functions working (witout to many errors. ... (What I have sent you is part my (digital) "black notebook" where I store my findings in the hope I find them someday)
For now some brain food out of (ui.js)
1. the convert function is called in the beginning and if everything goes well it returns either the new point only (ie. button state) or the `newPoint` AND the updated `fullDataset` (i.e chart)
now comes the tricky part
1. before emit is called the first time with the full dataDataset and stores the result. Here is where my if clause checks for the "fullDataset" property and returns that .
2. beforeEmit is called the SECOND time. This time with the newPoint. And this will go to the UI.
This makes sense as before emit is able to do mess around with the `fullDataset` and the `newPoint` separate. But to be honest the hole "theory of operations" is not 100% clear to me.
here the (reduced) part ui.js.
```auto
opt.node.on("input", function(msg) {
// Retrieve the dataset for this node
var oldValue = currentValues[opt.node.id];
// Call the convert function in the node to get the new value
// as well as the full dataset.
var conversion = opt.convert(msg.payload, oldValue, msg, opt.control.step);
// If the update flag is set, emit the newPoint, and store the full dataset
var fullDataset;
var newPoint;
if ((typeof(conversion) === 'object') && (conversion !== null) && (conversion.update !== undefined)) {
newPoint = conversion.newPoint;
fullDataset = conversion.updatedValues;
}
else if (conversion === undefined) {
fullDataset = oldValue;
newPoint = true;
}
else {
// If no update flag is set, this means the conversion contains
// the full dataset or the new value (e.g. gauges)
fullDataset = conversion;
}
// If we have something new to emit
if (newPoint !== undefined || !opt.emitOnlyNewValues || oldValue != fullDataset) {
currentValues[opt.node.id] = fullDataset;
// Determine what to emit over the websocket
// (the new point or the full dataset).
// Always store the full dataset.
var toStore = opt.beforeEmit(msg, fullDataset);
var toEmit;
if ((newPoint !== undefined) && (typeof newPoint !== "boolean")) { toEmit = opt.beforeEmit(msg, newPoint); }
else { toEmit = toStore; }
emitSocket(updateValueEventName, toEmit);
if (opt.persistantFrontEndValue === true) {
replayMessages[opt.node.id] = toStore;
}
}
});
```
> [@BartButenaers](#):
>
> Where the convert function is being called.
as described the convert callback is always called when a new message arrived by ui.js.
> [@BartButenaers](#):
>
> When $scope.\_data[0] is being filled
`$scope._data` is my array to hold the "ready to use" table for the uPlot. `_data[0][n]` is the time axes and `data[1..m][n]`are the rows. After `$scope.init(config)`is an empty array indicating that it is time to ask if the server holds `replayMessages`
What I dont understand is why the backend don't send the `replayMessages` after connects. I always get the last point only even with `persistantFrontEndValue: true`. So perhaps there is still a lack of knowledge on my side.
---
### Author:  [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)
#### Post date: [15 August 2021 16:45 UTC](https://discourse.nodered.org/t/problem-with-custom-ui-node-and-replaying-data/49478/16 "2021-08-15T16:45:40Z")
@Christian-Me,
Will need to have a look next week at your question, because not easy to follow on a smartphone...
Now I start to understand my confusion: in none of my UI nodes I have ever used a `node.on("input"...)`, because the dashboard has always handled the input msg processing fine for me. I only needed to implement the `beforeSend` and `beforeEmit` to get the job done...
So I assume you have removed the default dashboard input msg handler, and replaced it by your own one? And am I correct that you started with your handler as a clone from the default dashboard input msg handler, and you added functionality?
Or did I misunderstood this perhaps...
---
### Author:  [@Christian-Me](https://discourse.nodered.org/u/Christian-Me)
#### Post date: [17 August 2021 13:59 UTC](https://discourse.nodered.org/t/problem-with-custom-ui-node-and-replaying-data/49478/17 "2021-08-17T13:59:05Z")
Hope you had a nice holiday. (Or still on vacation- then ignore this until you are back and have time)
There is actually no real question any more. Think I understand the replay process now ([Problem with custom UI node and replaying data - #13 by Christian-Me](https://discourse.nodered.org/t/problem-with-custom-ui-node-and-replaying-data/49478/13))
The `node.on()` snippet is from under the hood of the dashboard (ui.js) to visualize the process= convert \> beforeEmit \> beforeEmit and how to get the full dataset on connect instead only the last message. I’m not messing with a separate event handler.
I expected that the dashboard would do this by itself if a fullDataset is present but never mind.
Currently I‘m optimizing the process to get the best performance with less traffic and dealing with sparse arrays and other fun stuff
But time is limited so progress is slow. Hope there is something to test soon.
---
### Author:  [@system](https://discourse.nodered.org/u/system)
#### Post date: [16 September 2021 13:59 UTC](https://discourse.nodered.org/t/problem-with-custom-ui-node-and-replaying-data/49478/18 "2021-09-16T13:59:09Z")
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.