Problem with custom UI node and replaying data

Hello,
I have a problem with replaying persistent data creating a custom chart ui-widget. After hours of debugging perhaps someone can help:

  • every message creates a new datapoint on the dashboard (works as expected)
  • I use the convert function to create the new datapoint conversion.newPoint
  • I push the new point to conversion.updatedValues
  • I set the conversion.updateflag to true

shorted Version

convert: function(value,fullDataset,msg,step) {
    var conversion = {
        updatedValues: [],
        newPoint: {},
    }
    if (fullDataset) conversion.updatedValues = fullDataset;
    conversion.newPoint.state = RED.util.getMessageProperty(msg, config.stateField || "payload");
    conversion.newPoint.topic = msg.topic;
    conversion.updatedValues.push(conversion.newPoint);
    conversion.update = true;
    return conversion;
},
  • then beforeEmit is called twice, once with the fullDataset I return and it is stored into replay messages and next with the newPoint. (I later found out I perhaps could have done with the beforeEmit function only)
beforeEmit: function(msg, fullDataset) {
    if (Array.isArray(fullDataset)) { return fullDataset };
    var newMsg = fullDataset;
   
    if (msg) {
        if (msg.socketid) newMsg.socketid = msg.socketid;
    }
    return { msg: newMsg };
},
  • using the debugger I can observe that replayMessages[nodeId] is holding the correct array of datapoints as expected

My problem is that the replayMessages array is never sent to the dashboard. Not on tab changes and not after page connects :frowning: Replaying the last message works but getting the historic data does not work.

Perhaps someone can shine some light on the correct mechanics of preparing, sending and replaying messages from the backend to the ui. Any working example (myLittleNode does not capture that topic and the original chart node seams not using the official API)

Hi @Christian-Me,
It is now some ago that I created a UI node, but indeed the msg replay was a battle every time...
I don't remember all the addWidget parameters anymore, but I assume you have been experimenting with those...
Perhaps you can share your addWidget parameters here, so everybody knows what you are using...
Bart

Hi Bart,

thank you offering your help :slight_smile: All based on myLittleUiNode (... working fine with my iro-color-picker, of which perhaps some fragments are still in the code) ...

                var done = ui.addWidget({                   // *REQUIRED* !!DO NOT EDIT!!
                    type: 'chart',
                    label: config.label,
                    tooltip: config.tooltip,
                    node: node,                             // *REQUIRED* !!DO NOT EDIT!!
                    order: config.order,                    // *REQUIRED* !!DO NOT EDIT!!
                    group: config.group,                    // *REQUIRED* !!DO NOT EDIT!!
                    width: config.width,                    // *REQUIRED* !!DO NOT EDIT!!
                    height: config.height,                  // *REQUIRED* !!DO NOT EDIT!!
                    format: html,                           // *REQUIRED* !!DO NOT EDIT!!
                    templateScope: "local",                 // *REQUIRED* !!DO NOT EDIT!!
                    emitOnlyNewValues: false,               // *REQUIRED* Edit this if you would like your node to only emit new values.
                    forwardInputMessages: false,  // *REQUIRED* Edit this if you would like your node to forward the input message to it's ouput.
                    storeFrontEndInputAsState: false,       // *REQUIRED* If the widget accepts user input - should it update the backend stored state ?

persistantFrontEndValue is missing but is true by default.
here I could trace it down into ui.js that the replay messages are stored:

           if (opt.persistantFrontEndValue === true) {
                replayMessages[opt.node.id] = toStore;
            }

attached the work in progress ui-uplot-charts.js.txt (15.8 KB)

It is already some time ago that I have been using those parameters...

Note that a tab switch works differently, because then the message will be replayed on the client side (i.e. in the dashboard)! See here for more info. While in the other cases the replaying is triggered by the server side of your node ... But that can perhaps explain why replaying the last message only adds a single point???

From another discussion I think that the message is stored in replaymessages (for the current node), as soon as the message arrives in your node. Do I understand correctly that you afterwards replace it by a full history of points (since the last msg only contained a single point?)??

My time is up for today ...

Think I found the problem and the solution! (nothing better than a Gin and Tonic and a relaxed debugging night session when the house is quiet :wink:

the property updatedValues must be an object and must contain a msg property ...

Something similar to this does the job

convert: function(value,fullDataset,msg,step) {
    var conversion = {
        updatedValues: {msg:{payload:[]}},
        newPoint: {},
    }
    if (fullDataset) conversion.updatedValues = fullDataset;
    conversion.newPoint.state = RED.util.getMessageProperty(msg, config.stateField || "payload");
    conversion.newPoint.topic = msg.topic;
    conversion.updatedValues.msg.payload.push({conversion.newPoint.topic:conversion.newPoint.state});
    conversion.update = true;
    return conversion;
},

as so often we (including myself) are particular bad in naming things. Instead updatedValues perhaps something like updatedMessages or perhaps better replayMessage could have given me the right clue and saved some time digging into socket.js :wink:

1 Like

Ok I solved the reconnect problem = I get the accumulated dataset from the backend ... but the tab changed problem is still an issue.

After a tab change I get the last message replayed ... fine, but the accumulated data (I collected while widget was visible) in $scope._data is gone :frowning: I any way this is ok as over time the tab was not visible new datapoints could have arrived, so I need the up to date data from the backend.

Question is : How can I let the client ask for the updatedValues.msg after a tab change instead getting only the last message (newPoint).

I could send always the complete data set every time but this would be a waste of bandwidth (one intention to use uPlot is to be capable of huge datasets and fast updates).

As you can see here, we used a trick to see whether a message is being replayed on the client side (e.g. by a tab switch): add a field to the input msg (i.e. on the client side copy of the input msg). When that field is already available when the watch is triggered, then you know that it is a client-side replayed msg (which has previously been handled already by this watch...). In that case you could get your complete dataset from the server.
But of course it is only a trick...

Hmmm perhaps I need a G&T :crazy_face:

I'm aware of "rebouncing" messages .... Your trick is nice (and better than my solution in iro-color), but I think this is not my problem as the widget does not emit messages (display only).

On Tab changes I only get the last datapoint ... but I need in this case the complete dataset to (re-)initialize the chart.
On connect get exactly the message I need and easy to identify as msg.fullDataset
Tab changes after a connect (with no later updates) are fine as the last message has the full dataset.

But how I can get the backend to send me the full dataset (instead the last datapoint)?

Now I'm lost. How can you get messages (and replayed messages) in the dashboard side, when your node does not emit messages, i.e. push messages from server-side to client-side (after calling your beforeEmit function)?

Sorry, I was unclear ... the widget does not emit messages (òutputs=0)
Internally the node (backend) emits messages to the client (frontend).

Yes in the beginning I was struggling with the names:

  • The ui node emits (=pushes) messages via socketio from server to client. In the server you can update the messages in ´beforeEmit´, and in the client you can ´watch´ them.
  • The ui node sends messages on the server side to its output, and you can update them in ´beforeSend´.

I "think" you cannot push them from the server to the client, since the server-side of your node is not aware of this happening. But I might be mistaken!! Hopfully somebody else can confirm or deny this...

That would lean that the client needs to get the data from the server. For example when the watch sees a replayed message, it:

  • calls an endpoint of your node
  • sends a msg to the server (e.g. topic="getfulldata"), which you handle in beforeSend by pushing the full data msg to the client

Not sure if my proposals make sense, because I haven't tried it!

1 Like

That sounds like a pretty good concept to me! Thank you! Will walk the :dog2: grab me a :cocktail:and give it a try. Already see some challenges

  • distinguish a tab change from a normal reconnect
  • then send the complete dataset to a specific socketid only

But I also remember something similar I found in the original chart node (think it was a onTabChange event listener I could not get working in the first try as the api is different)

So some options to play with :+1:

1 Like

After a while of trying to emit a message directly (failing to get access to emitSocket() I found out that is is possible to send a message to myself via node.receive()

For anybody interested or anybody saying I'm doing it all wrong here are some code fragments:

  1. the $watch code:
                        $scope.$watch('msg', function(msg) {
                            if (!msg) { return; } // Ignore undefined msg
                            if ($scope._data[0]===undefined) {
                                console.log('uPlot first Message > asking for replay');
                                $scope.send({payload:"R"});
                                return;
                            }
                            if (msg.hasOwnProperty('fullDataset')) {
                                console.log('uPlot fullDataset received:',msg);
                                
                                // do something with the full dataset 

                                });
                            } else if (isNewDataPoint(msg)){
                                console.log('uPlot dataPoint received:',msg);

                                // do something with the datapoint
 
                            }
                        });
  1. in beforeSend trigger let's trigger a message to myself
                    beforeSend: function (msg, orig) {
                        if (orig.msg.payload === 'R') {
                            node.receive(orig.msg);
                            return;
                        }
                        if (orig) {
                            var newMsg = {};

                            // do something here if your node emits data
                        }
                    },
  1. in convert return the full dataset if a replay is requested
                    convert: function(value,fullDataset,msg,step) {
                        if (msg.payload==='R') {
                            return fullDataset;
                        }
                        var conversion = {
                            updatedValues: {
                                msg:{
                                    fullDataset : []
                                }},
                            newPoint: {},
                        }

                        // do your converting business

                        return conversion;
                    },
  1. in beforeEmit return the full dataset again (if present and well formed)
                    beforeEmit: function(msg, fullDataset) {
                        if (fullDataset && fullDataset.hasOwnProperty('msg') && fullDataset.msg.hasOwnProperty('fullDataset')) {
                            return fullDataset 
                        };
                        
                        // prepare your payload ready for launch
                        
                        return { msg: newMsg };
                    },

and then you should get something like this if you switch the tab or refresh your page:

later messages simply add a new datapoint:

Let's see if this solution works in a longer development and debugging period

and all of this will later perhaps look like this (just a proof of concept and my own capabilities)

Hi @Christian-Me,
Sorry for the lare reply!
Thanks for sharing the code sippets, which summarize nicely your mechanism! But can you also share a link to your repo (if available), because some things are not clear to me:

  • When I see function(msg, fullDataset) it looks to me you always send a full dataset to the frontend. But that is not the case I assume?
  • Where the convert function is being called.
  • When $scope._data[0] is being filled

Hi @BartButenaers

Will do as soon I cleaned up some of the mess and get the basic functions working (witout to many errors. ... (What I have sent you is part my (digital) "black notebook" where I store my findings in the hope I find them someday)

For now some brain food out of (ui.js)

  1. the convert function is called in the beginning and if everything goes well it returns either the new point only (ie. button state) or the newPoint AND the updated fullDataset (i.e chart)

now comes the tricky part

  1. before emit is called the first time with the full dataDataset and stores the result. Here is where my if clause checks for the "fullDataset" property and returns that .

  2. beforeEmit is called the SECOND time. This time with the newPoint. And this will go to the UI.

This makes sense as before emit is able to do mess around with the fullDataset and the newPoint separate. But to be honest the hole "theory of operations" is not 100% clear to me.

here the (reduced) part ui.js.

    opt.node.on("input", function(msg) {
        // Retrieve the dataset for this node
        var oldValue = currentValues[opt.node.id];

        // Call the convert function in the node to get the new value
        // as well as the full dataset.
        var conversion = opt.convert(msg.payload, oldValue, msg, opt.control.step);

        // If the update flag is set, emit the newPoint, and store the full dataset
        var fullDataset;
        var newPoint;
        if ((typeof(conversion) === 'object') && (conversion !== null) && (conversion.update !== undefined)) {
            newPoint = conversion.newPoint;
            fullDataset = conversion.updatedValues;
        }
        else if (conversion === undefined) {
            fullDataset = oldValue;
            newPoint = true;
        }
        else {
            // If no update flag is set, this means the conversion contains
            // the full dataset or the new value (e.g. gauges)
            fullDataset = conversion;
        }
        // If we have something new to emit
        if (newPoint !== undefined || !opt.emitOnlyNewValues || oldValue != fullDataset) {
            currentValues[opt.node.id] = fullDataset;

            // Determine what to emit over the websocket
            // (the new point or the full dataset).
            // Always store the full dataset.
            var toStore = opt.beforeEmit(msg, fullDataset);
            var toEmit;
            if ((newPoint !== undefined) && (typeof newPoint !== "boolean")) { toEmit = opt.beforeEmit(msg, newPoint); }
            else { toEmit = toStore; }

            emitSocket(updateValueEventName, toEmit);
            if (opt.persistantFrontEndValue === true) {
                replayMessages[opt.node.id] = toStore;
            }
        }
    });

as described the convert callback is always called when a new message arrived by ui.js.

$scope._data is my array to hold the "ready to use" table for the uPlot. _data[0][n] is the time axes and data[1..m][n]are the rows. After $scope.init(config)is an empty array indicating that it is time to ask if the server holds replayMessages

What I dont understand is why the backend don't send the replayMessages after connects. I always get the last point only even with persistantFrontEndValue: true. So perhaps there is still a lack of knowledge on my side.

@Christian-Me,
Will need to have a look next week at your question, because not easy to follow on a smartphone...

Now I start to understand my confusion: in none of my UI nodes I have ever used a node.on("input"...), because the dashboard has always handled the input msg processing fine for me. I only needed to implement the beforeSend and beforeEmit to get the job done...

So I assume you have removed the default dashboard input msg handler, and replaced it by your own one? And am I correct that you started with your handler as a clone from the default dashboard input msg handler, and you added functionality?

Or did I misunderstood this perhaps...

Hope you had a nice holiday. (Or still on vacation- then ignore this until you are back and have time)

There is actually no real question any more. Think I understand the replay process now (Problem with custom UI node and replaying data - #13 by Christian-Me)
The node.on() snippet is from under the hood of the dashboard (ui.js) to visualize the process= convert > beforeEmit > beforeEmit and how to get the full dataset on connect instead only the last message. I’m not messing with a separate event handler.
I expected that the dashboard would do this by itself if a fullDataset is present but never mind.

Currently I‘m optimizing the process to get the best performance with less traffic and dealing with sparse arrays and other fun stuff
But time is limited so progress is slow. Hope there is something to test soon.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.