then beforeEmit is called twice, once with the fullDataset I return and it is stored into replay messages and next with the newPoint. (I later found out I perhaps could have done with the beforeEmit function only)
beforeEmit: function(msg, fullDataset) {
if (Array.isArray(fullDataset)) { return fullDataset };
var newMsg = fullDataset;
if (msg) {
if (msg.socketid) newMsg.socketid = msg.socketid;
}
return { msg: newMsg };
},
using the debugger I can observe that replayMessages[nodeId] is holding the correct array of datapoints as expected
My problem is that the replayMessages array is never sent to the dashboard. Not on tab changes and not after page connects Replaying the last message works but getting the historic data does not work.
Perhaps someone can shine some light on the correct mechanics of preparing, sending and replaying messages from the backend to the ui. Any working example (myLittleNode does not capture that topic and the original chart node seams not using the official API)
Hi @Christian-Me,
It is now some ago that I created a UI node, but indeed the msg replay was a battle every time...
I don't remember all the addWidget parameters anymore, but I assume you have been experimenting with those...
Perhaps you can share your addWidget parameters here, so everybody knows what you are using...
Bart
thank you offering your help All based on myLittleUiNode (... working fine with my iro-color-picker, of which perhaps some fragments are still in the code) ...
var done = ui.addWidget({ // *REQUIRED* !!DO NOT EDIT!!
type: 'chart',
label: config.label,
tooltip: config.tooltip,
node: node, // *REQUIRED* !!DO NOT EDIT!!
order: config.order, // *REQUIRED* !!DO NOT EDIT!!
group: config.group, // *REQUIRED* !!DO NOT EDIT!!
width: config.width, // *REQUIRED* !!DO NOT EDIT!!
height: config.height, // *REQUIRED* !!DO NOT EDIT!!
format: html, // *REQUIRED* !!DO NOT EDIT!!
templateScope: "local", // *REQUIRED* !!DO NOT EDIT!!
emitOnlyNewValues: false, // *REQUIRED* Edit this if you would like your node to only emit new values.
forwardInputMessages: false, // *REQUIRED* Edit this if you would like your node to forward the input message to it's ouput.
storeFrontEndInputAsState: false, // *REQUIRED* If the widget accepts user input - should it update the backend stored state ?
persistantFrontEndValue is missing but is true by default.
here I could trace it down into ui.js that the replay messages are stored:
if (opt.persistantFrontEndValue === true) {
replayMessages[opt.node.id] = toStore;
}
It is already some time ago that I have been using those parameters...
Note that a tab switch works differently, because then the message will be replayed on the client side (i.e. in the dashboard)! See here for more info. While in the other cases the replaying is triggered by the server side of your node ... But that can perhaps explain why replaying the last message only adds a single point???
From another discussion I think that the message is stored in replaymessages (for the current node), as soon as the message arrives in your node. Do I understand correctly that you afterwards replace it by a full history of points (since the last msg only contained a single point?)??
as so often we (including myself) are particular bad in naming things. Instead updatedValues perhaps something like updatedMessages or perhaps better replayMessage could have given me the right clue and saved some time digging into socket.js
Ok I solved the reconnect problem = I get the accumulated dataset from the backend ... but the tab changed problem is still an issue.
After a tab change I get the last message replayed ... fine, but the accumulated data (I collected while widget was visible) in $scope._data is gone I any way this is ok as over time the tab was not visible new datapoints could have arrived, so I need the up to date data from the backend.
Question is : How can I let the client ask for the updatedValues.msg after a tab change instead getting only the last message (newPoint).
I could send always the complete data set every time but this would be a waste of bandwidth (one intention to use uPlot is to be capable of huge datasets and fast updates).
As you can see here, we used a trick to see whether a message is being replayed on the client side (e.g. by a tab switch): add a field to the input msg (i.e. on the client side copy of the input msg). When that field is already available when the watch is triggered, then you know that it is a client-side replayed msg (which has previously been handled already by this watch...). In that case you could get your complete dataset from the server.
But of course it is only a trick...
I'm aware of "rebouncing" messages .... Your trick is nice (and better than my solution in iro-color), but I think this is not my problem as the widget does not emit messages (display only).
On Tab changes I only get the last datapoint ... but I need in this case the complete dataset to (re-)initialize the chart.
On connect get exactly the message I need and easy to identify as msg.fullDataset
Tab changes after a connect (with no later updates) are fine as the last message has the full dataset.
But how I can get the backend to send me the full dataset (instead the last datapoint)?
Now I'm lost. How can you get messages (and replayed messages) in the dashboard side, when your node does not emit messages, i.e. push messages from server-side to client-side (after calling your beforeEmit function)?
Yes in the beginning I was struggling with the names:
The ui node emits (=pushes) messages via socketio from server to client. In the server you can update the messages in ´beforeEmit´, and in the client you can ´watch´ them.
The ui node sends messages on the server side to its output, and you can update them in ´beforeSend´.
I "think" you cannot push them from the server to the client, since the server-side of your node is not aware of this happening. But I might be mistaken!! Hopfully somebody else can confirm or deny this...
That would lean that the client needs to get the data from the server. For example when the watch sees a replayed message, it:
calls an endpoint of your node
sends a msg to the server (e.g. topic="getfulldata"), which you handle in beforeSend by pushing the full data msg to the client
Not sure if my proposals make sense, because I haven't tried it!
That sounds like a pretty good concept to me! Thank you! Will walk the grab me a and give it a try. Already see some challenges
distinguish a tab change from a normal reconnect
then send the complete dataset to a specific socketid only
But I also remember something similar I found in the original chart node (think it was a onTabChange event listener I could not get working in the first try as the api is different)
After a while of trying to emit a message directly (failing to get access to emitSocket() I found out that is is possible to send a message to myself via node.receive()
For anybody interested or anybody saying I'm doing it all wrong here are some code fragments:
the $watch code:
$scope.$watch('msg', function(msg) {
if (!msg) { return; } // Ignore undefined msg
if ($scope._data[0]===undefined) {
console.log('uPlot first Message > asking for replay');
$scope.send({payload:"R"});
return;
}
if (msg.hasOwnProperty('fullDataset')) {
console.log('uPlot fullDataset received:',msg);
// do something with the full dataset
});
} else if (isNewDataPoint(msg)){
console.log('uPlot dataPoint received:',msg);
// do something with the datapoint
}
});
in beforeSend trigger let's trigger a message to myself
beforeSend: function (msg, orig) {
if (orig.msg.payload === 'R') {
node.receive(orig.msg);
return;
}
if (orig) {
var newMsg = {};
// do something here if your node emits data
}
},
in convert return the full dataset if a replay is requested
convert: function(value,fullDataset,msg,step) {
if (msg.payload==='R') {
return fullDataset;
}
var conversion = {
updatedValues: {
msg:{
fullDataset : []
}},
newPoint: {},
}
// do your converting business
return conversion;
},
in beforeEmit return the full dataset again (if present and well formed)
beforeEmit: function(msg, fullDataset) {
if (fullDataset && fullDataset.hasOwnProperty('msg') && fullDataset.msg.hasOwnProperty('fullDataset')) {
return fullDataset
};
// prepare your payload ready for launch
return { msg: newMsg };
},
and then you should get something like this if you switch the tab or refresh your page:
Hi @Christian-Me,
Sorry for the lare reply!
Thanks for sharing the code sippets, which summarize nicely your mechanism! But can you also share a link to your repo (if available), because some things are not clear to me:
When I see function(msg, fullDataset) it looks to me you always send a full dataset to the frontend. But that is not the case I assume?
Will do as soon I cleaned up some of the mess and get the basic functions working (witout to many errors. ... (What I have sent you is part my (digital) "black notebook" where I store my findings in the hope I find them someday)
For now some brain food out of (ui.js)
the convert function is called in the beginning and if everything goes well it returns either the new point only (ie. button state) or the newPoint AND the updated fullDataset (i.e chart)
now comes the tricky part
before emit is called the first time with the full dataDataset and stores the result. Here is where my if clause checks for the "fullDataset" property and returns that .
beforeEmit is called the SECOND time. This time with the newPoint. And this will go to the UI.
This makes sense as before emit is able to do mess around with the fullDataset and the newPoint separate. But to be honest the hole "theory of operations" is not 100% clear to me.
here the (reduced) part ui.js.
opt.node.on("input", function(msg) {
// Retrieve the dataset for this node
var oldValue = currentValues[opt.node.id];
// Call the convert function in the node to get the new value
// as well as the full dataset.
var conversion = opt.convert(msg.payload, oldValue, msg, opt.control.step);
// If the update flag is set, emit the newPoint, and store the full dataset
var fullDataset;
var newPoint;
if ((typeof(conversion) === 'object') && (conversion !== null) && (conversion.update !== undefined)) {
newPoint = conversion.newPoint;
fullDataset = conversion.updatedValues;
}
else if (conversion === undefined) {
fullDataset = oldValue;
newPoint = true;
}
else {
// If no update flag is set, this means the conversion contains
// the full dataset or the new value (e.g. gauges)
fullDataset = conversion;
}
// If we have something new to emit
if (newPoint !== undefined || !opt.emitOnlyNewValues || oldValue != fullDataset) {
currentValues[opt.node.id] = fullDataset;
// Determine what to emit over the websocket
// (the new point or the full dataset).
// Always store the full dataset.
var toStore = opt.beforeEmit(msg, fullDataset);
var toEmit;
if ((newPoint !== undefined) && (typeof newPoint !== "boolean")) { toEmit = opt.beforeEmit(msg, newPoint); }
else { toEmit = toStore; }
emitSocket(updateValueEventName, toEmit);
if (opt.persistantFrontEndValue === true) {
replayMessages[opt.node.id] = toStore;
}
}
});
as described the convert callback is always called when a new message arrived by ui.js.
$scope._data is my array to hold the "ready to use" table for the uPlot. _data[0][n] is the time axes and data[1..m][n]are the rows. After $scope.init(config)is an empty array indicating that it is time to ask if the server holds replayMessages
What I dont understand is why the backend don't send the replayMessages after connects. I always get the last point only even with persistantFrontEndValue: true. So perhaps there is still a lack of knowledge on my side.
@Christian-Me,
Will need to have a look next week at your question, because not easy to follow on a smartphone...
Now I start to understand my confusion: in none of my UI nodes I have ever used a node.on("input"...), because the dashboard has always handled the input msg processing fine for me. I only needed to implement the beforeSend and beforeEmit to get the job done...
So I assume you have removed the default dashboard input msg handler, and replaced it by your own one? And am I correct that you started with your handler as a clone from the default dashboard input msg handler, and you added functionality?
Hope you had a nice holiday. (Or still on vacation- then ignore this until you are back and have time)
There is actually no real question any more. Think I understand the replay process now (Problem with custom UI node and replaying data - #13 by Christian-Me)
The node.on() snippet is from under the hood of the dashboard (ui.js) to visualize the process= convert > beforeEmit > beforeEmit and how to get the full dataset on connect instead only the last message. I’m not messing with a separate event handler.
I expected that the dashboard would do this by itself if a fullDataset is present but never mind.
Currently I‘m optimizing the process to get the best performance with less traffic and dealing with sparse arrays and other fun stuff
But time is limited so progress is slow. Hope there is something to test soon.
Replaying the last message works but getting the historic data does not work.
All based on myLittleUiNode (... working fine with my iro-color-picker, of which perhaps some fragments are still in the code) ...
grab me a 
and give it a try. Already see some challenges
