How to sync the FULL config between server and client side state

Dashboard
,
1

Hi folks,

In the old AngularJs dashboard there was a mechanism called msg replay, which meant that the last input message was send again to the client side widget (e.g. when the browser page was refreshed). That way you could reconfigure your widget again, to match the last state stored in that last input message.

While that sounded at first a very brilliant idea, it resulted in a tremendous headache for ui node developers. For the simple reason that the last message only contains a small fraction of the widget full state. Let's explain this by a hypothetical button example:

  1. Message is injected to change the button label.
  2. Message is injected to change the button color.
  3. The browser page is refreshed.
  4. The last messages is resend, so you set the correct button color. However you don't have the correct button label, because the last message contains only a part of the state (i.e. the last message only the color, but not the label).

Thorsten had solved that in Flexdash, by having messages updating a server-side state and sending the delta state to the clients (to update the client-side state to keep it in sync with the server-side state). Same hypothetical button example:

  1. Message is injected to change the button label.
  2. The server-side state is updated with the new label and the new label is send to the clients, which update their client-side state with the new label.
  3. Message is injected to change the button color.
  4. The server-side state is updated with the new color, and the new color is send to the clients, which update their client-side state with the new color. Both server-side and client-side states are in sync and contain the FULL state.
  5. The browser page is refreshed.
  6. The server-side is resend, so the client-side state is correctly updated with both the new label and the new color.

For my old AngularJs ui-svg node it was the worst. For example in a floorplan you visualize a lot of devices, so a lot of state updates are required. Users had to find their own workarounds, e.g. by replaying a queue of N messages. But that is totally not waterproof, because:

  • If N is not large enough, then you loose old state updates.
  • The queue contains lot of messages overriding each other values over time (e.g. color red -> color green -> color red -> color green -> ...).
  • Very inefficient to replay lots of old message, containing most of the time obsolete state.

In an attempt to avoid a new message replay mechanism in the new dashboard, I have at the time being tried to explain this problem in this issue. However the message based approach has been reintroduced again, so unfortunately I am completely stuck again.

So I starting rereading the documentation again, to see if there is a way to solve my issue. But not sure how I can workaround it when reading this part of the documentation:

image
image737Ă—369 35.1 KB

I think I have overlooked something. Because the statement in most cases a widget only needs reference to the last message only makes sense when that last message contains the full state. I read somewhere in the documentation that the (recently introduced) dynamic properties are being merged automatically into the static properties from the config screen. So perhaps that is a new dashboard change to improve the message replay mechanism? Perhaps the keyword "message" in the client-side context is referring to messages from server to client, and not the input messages of the ui node? No idea...

Would appreciate a lot if somebody could enlighten me! Because I don't see the overal picture anymore.

Thanks!!
Bart

2

I have to confess - read through this, I'm now confused myself. Will zip through relevant docs and codebase and make sure i present a clear picture of what's going on.

I do believe though, that, as you say, we have the shortfall of last msg being replayed rather than an accumulative state per FlexDash - always open to change/iteration though, and it wouldn't be too much of a breaking change (if at all) to introduce this instead

1 Like
3

Adding my $0.02:
I believe we should not be saving msg sequences, as some messages represent actions, not data updates, and you would not want to replay these. For example, you may want to periodically send a message to your node telling it to email its data, and you do not want to replay this upon reload.

IMHO, the datastore should maintain a data image which enables the node to restore itself upon open/refresh, and look the same as in other clients. This data image can also include history (e.g. for a chart - an array of data updates), but in any case this would be a node-specific implementation, not generic blind replay of messages.

1 Like
4

TLDR: statestore is for overriding Node-RED config, and the updated (including dynamic updates) config is sent on the ui-config event. The datastore tracks msg objects, generally only the last received message, but can support a history of messages if required using datastore.append in the case of charts (as an example).

Server-Side Stores

Docs: State Management | Node-RED Dashboard 2.0

  • datastore: A map of each widget to the latest msg received by a respective node in the Editor.
  • statestore: A store for all dynamic properties set on widgets (e.g. visibility or setting a property at runtime). Often, these values are overrides of the base configuration defined in Node-RED. (This is our equivalent of FlexDash's state store.)

The pattern we've implemented in core is that the statestore is populated when a node receives messages, and detects a property in that message that it is interested in, e.g in ui-button.js:

Loading Events

Docs: Events Architecture | Node-RED Dashboard 2.0

UI Config/State

When the Dashboard is first loaded on a page refresh or navigation, we send a ui-config event, which details every widget, page, theme, etc. This contains the specification for each node (which is the Node-RED config schema), but with the relevant values overriden by the widget's statestore.

This is the single source of truth for the state of a widget at that time.

When building your own node, in the Node-RED .js file, copy the pattern seen in the buttons above. To access the stores from within a third-party node, you can call:

const group = RED.nodes.getNode(config.group)
const base = group.getBase()
base.stores.<data/state>

I don't have an example in ui-example node for dynamic properties yet, so I'll try and get something added.

Anything you put into the statestore will be merged with the config of your node and transmitted for ui-config.

Widget Load

For each widget, when that itself loads, we send a widget-load(msg) event, which passes the latest received msg from the datastore.

The widget does not need to do anything with this if it's not appropriate. The dynamic/updated state would already have been sent to the widget from the statestore via the merged ui-config schema.

Updates Required

  • I had a mis-typed statement in the statestore description, which claimed " Often, these values are overrides of the base configuration found in the datastore.". This is not true, and I corrected above, it should read: "Often, these values are overrides of the base configuration defined in Node-RED. "
  • On my side, I do need to update the "Input" events architecture diagram to show an example of the Dynamic Properties approach we now have

Ideas for Improvement:

Alongside the msg being sent in widget-load, I could also pass the config? It's already sent with the ui-config event, but for peace of mind, it's a lightweight addition, that makes sense to include?

1 Like
5

Taking the opportunity to remind that the on-widget event should be sent (with an empty message) even when there is no data in the datastore ("no data" is also data).

Are you referring to the node's config object with the configured node properties? Is it different than what is uploaded into the this.props object?

6

They are the same. this.props was poor variable naming on my part, that, to change now would cause a lot of breaking changes

1 Like
7

On the original D1 dashboard the replay was only for the payload part of the msg, as a) that was the original design and the concept of allowing dynamic update of things like labels was only added much later, and b) extending it to the whole msg would potentially mean storing masses of data - (or how would you select which properties to store, etc)

1 Like
8

In third party nodes, at least, I do not believe that is necessarily the case. What is stored in the datastore is entirely up to the developer. In the example node nodes/ui-example.js the onInput function is

            onInput: function (msg, send, done) {
                // store the latest value in our Node-RED datastore
                base.stores.data.save(base, node, msg)
                // send it to any connected nodes in Node-RED
                send(msg)
            },

This stores the latest message in the store and then sends it to the client, so the store does contain the last message, and when a node connects only that latest message is sent to the node.

In my nodes I have not done that, I use the datastore as a store for the complete state of the widget. My classic gauge node, for example, requires current values for each needle, but those values arrive individually, identified by topic. When a needle value is received I update the value for that needle in the store, and then send the original message to the client.
When a client connects it gets sent the datastore, which contains the full state and saves that locally. When a new message is received, for example with a needle value, it updates that needle value in its local store. The result is that the server and client always have a full copy of the state. So far this technique is working well for me and is very simple to understand and implement. So far I am not aware of any issues with using this simple system.

Edit: I have made a slight amendment to the above, adding the word 'original' for clarification in the sentence 'When a needle value is received I update the value for that needle in the store, and then send the original message to the client'.

1 Like
9

Yep, good point - I can't control what people do in their third-party nodes. I've built in the APIs to copy core patterns, but if devs want to go off-piste, then that's fine too

1 Like
10

The second part is only true if the application is a single page/Tab. If we have several tabs and are visualizing another tab, all the configuration that had been sent to the widget will be lost (except the last one) when we return the tab where the widget is (no reload, just switching tabs).

This is related to a an PR I have open, in that case related to UI-Button but it is applicable to all widgets. One of the changes was in the widget-load event adding to the message all the Dynamic Properties that have been received for that Widget, in order to restore the widget to the correct state even if it had received several separated messages each one with one Dynamic Property set...

For this to work is also necessary to add a onLoad event in the widget itself, some of them only have the onDynamicProperties event which is not triggered on widget-load.

The is also the necessity to also sync other things that are not the dynamic properties, like the enable, class and visible, depending on the widget may be more, that at the moment with the latest updates are not part of Dynamic Propreties and are left a little in the limbo. These properties should pass to the Dynamic Properties (ui_update) group, at least the enable that is applicable to the widget itself, class and visible seems more to the nrdb-ui-widget (row element) than the widget (button) itself.

1 Like
11

Do you see any issues with my strategy, which to me seems much simpler than the core pattern? For example, in the chart node, I assume that the state store retains an array of a (potentially) large number of previous messages so that the client can build the chart on initial connection. Would it not be much simpler to understand if the server just maintained a copy of the current state of the chart?

Similarly with the issue @arturv2000 notes regarding enable etc. If those are maintained as part of the state in the server then the problem is solved.

1 Like
12

That's valid, which is why I propose including config inside widget-load event

1 Like
13

Yes, it's a difficult one as I need to handle backward compatibility, whilst also offering improvements to the API

1 Like
14

I have just re-read the earlier posts again and realised that the statestore is actually being used in the way that I have used the datastore, so in fact I should put all my dynamic properties (such as gauge needle current values) into the statestore, not the datastore. I think that means that actually I don't have any need for the datastore. In fact I am not sure why it is needed at all.

With ui_update data, in a third party node, will ui_update values automatically get included in the statestore, overriding the config, or do I need to handle that myself?

1 Like
15

I think you mean datastore here, as you just mentioned you'd be switching stuff too the statestore.

The datastore is still needed because in multi-tenant use cases, when we replay messages, how do we know who should receive that message? We need the full msg context to be sure of this.

I'd separated out state and data intentionall, where state defines anything in the node's configuration, the data is the latest msg objects, and generally useful to access msg.payload, but not limited to.

It's a practice we've implemented, up to third-party devs to follow suit, or do their own thing

1 Like
16

Ok, thanks everybody!
Will try to digest the great feedback from everybody this weekend.
I went back to the drawing table, and indeed it looks quite how I had the Flexdash setup in my mind. More as I had expected to be honest.
I need to read again through some other discussion (like e.g. the one I had last week with @Colin about his experience), and then I will get back here.
I need to visualize it in a diagram, to allow my brain to understand all of it...

17

Yes indeed, thanks, I have corrected my post.

Is the latest message saved for each user in the multi-tenant case? Otherwise it doesn't really work does it? Only one user will have a message to be replayed.

1 Like
18

This question probably implies a lack of understanding on my part. I understand this is about pre-built widgets but I was wondering if any of this is / can be utilised in a ui-template widget?

19

I created quickly a first DRAFT version, and share it now already because I have still a lot of questions in RED. If somebody can answer some of those, that would be nice:

image
image1600Ă—810 167 KB

Some general remarks:

  1. I "think" that I was partly confused because the full state of a widget is called also "msg" in the frontend, but not sure.
  2. When a message is loaded then the last input msg is being replayed. Imho I certainly would not add extra parameters to the on-load event, but skip the input msg from that event completely. Because that msg:
    • Contains only partial state, so that is looking for troubles.
    • Mixing in the frontend state that arrives from input messages and from the servers-side stores is VERY confusing.
  3. Like @Colin mentioned, it is not clear to me how you support multi-tenant stuff by replaying only the last input message, containing partial state.
  4. Lots more questions in my drawing.

If anybody wants to draw on top of my drawing in Paint, be my guest. I will adjust my drawing with the feedback during the weekend.

Thanks!!!!!!!!!!!!!!!

20

Following my diagram, I will try to avoid using the Data store at all cost.

So I want to store my dynamic properties (from msg.ui_update) into the State store. However that fails, because only the Data store seems to be available:

image
image724Ă—391 19.3 KB

EDIT: I have registered a Github issue and will try to solve it this weekend.