# How to sync the FULL config between server and client side state

**URL:** <https://discourse.nodered.org/t/how-to-sync-the-full-config-between-server-and-client-side-state/89252>\
**Category:** Dashboard\
**Tags:** flexdash, dashboard-2\
**Created:** [4 July 2024 05:55 UTC](https://discourse.nodered.org/t/how-to-sync-the-full-config-between-server-and-client-side-state/89252 "2024-07-04T05:55:31Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [5 July 2024 15:59 UTC](https://discourse.nodered.org/t/how-to-sync-the-full-config-between-server-and-client-side-state/89252/21 "2024-07-05T15:59:19Z")

</div>

I will have a look at your drawing in detail, but I am tied up for a few days so I may not be able to provide feedback promptly.

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [6 July 2024 07:51 UTC](https://discourse.nodered.org/t/how-to-sync-the-full-config-between-server-and-client-side-state/89252/22 "2024-07-06T07:51:12Z")

</div>

> [@Colin](#):
>
> I will have a look at your drawing in detail, but I am tied up for a few days

No hurry Colin. I had a look at some dashboard code, which provided me some more insights in how it currently works. Will need to update my diagram with all this information, so we can continue this discussion.

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [8 July 2024 21:24 UTC](https://discourse.nodered.org/t/how-to-sync-the-full-config-between-server-and-client-side-state/89252/23 "2024-07-08T21:24:04Z")

</div>

> [@BartButenaers](#):
>
> Will need to update my diagram with all this information, so we can continue this discussion.

Ok, I am back from the drawing table.  
The fog has a little bit cleared in my head.

Although I have limited my drawing to only the data streams in the dashboard, unfortunately it contains much more arrows as I had hoped. But I really need to see everything that is going on behind the scenes, in order to understand how it works. And all those arrows are in fact the reason why I had to draw it, in order to support my brain to understand how it works...

Here is a (large) printscreen:

 ![dashboard_draft1](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/0/c/0c5d9ee991c2366e9980a2e5196618a7d30191e0.png)

And here is the corresponding drawing file, in case anybody wants to open it (in [https://excalidraw.com/](https://excalidraw.com/)). Note that I had to change the extension from .excalidraw to .txt because it was rejected by Discourse: [node\_red\_dashboard\_3.txt](https://discourse.nodered.org/uploads/short-url/wyhPCK0NEsEmnsfRHSe2XCugulJ.txt) (413.9 KB)

My questions and proposals are all in _ **red** _. I would appreciate if somebody could take the time to review it, in order to make this a useful diagram for UI node developers and dashboard contributors.

When looking at this diagram, it is unfortunately still not clear to me why we need a _ **datastore** _ to store messages (which are replayed afterwards) while the _ **statestore** _ already does a good job to store the state. The datastores and related message replaying makes the development pretty complex to understand.

Hopefully my feedback isn't interpreted as negative, because dashboard D2 has become a piece of art in a small amount of time. And I am still a big fan of it! But imho the message-based approach does really increase the threshold to start contributing.

Thanks for reading!!!  
Bart

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [9 July 2024 15:44 UTC](https://discourse.nodered.org/t/how-to-sync-the-full-config-between-server-and-client-side-state/89252/24 "2024-07-09T15:44:26Z")

</div>

![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/1/a/1a6a29fbeed0f3b15b63d4ff245a69a28af858d7.png)

Only to the newly connected client.  
I am still confused about some of this though. As I understand it, on connection, the complete state of _all_ widgets is sent in the ui-config object, and if the developer tools is enabled this can be seen. However, for third party nodes at least, this is not made directly available. What is made available is the `mounted` function which is given `this.props` which is the node's config merged with the datastore.

![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/7/5/75d94619158cc494b1154cbf186b3c37f773277c.png)

i don't think there is any check on check on `msg._client` for the update of the state store. I may be wrong here though.

That is as far as I have got at the moment.

> [@BartButenaers](#):
>
> it is unfortunately still not clear to me why we need a _ **datastore** _ to store messages (which are replayed afterwards) while the _ **statestore** _ already does a good job to store the state.

Ditto. In any new nodes I am working on I am, for the moment, ignoring the data store, and not writing incoming messages to it, so I do not get the last message replayed on connection. Everything I need is in the state store.

**[Edit]** At least I would do, but at the moment the state store is not available to third party nodes, so I am having to use just the data store instead, but using it as a state store, not saving messages to it.

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [9 July 2024 16:23 UTC](https://discourse.nodered.org/t/how-to-sync-the-full-config-between-server-and-client-side-state/89252/25 "2024-07-09T16:23:28Z")

</div>

@Colin,  
Thanks for reviewing! Really appreciated!!

> [@Colin](#):
>
> i don't think there is any check on check on `msg._client` for the update of the state store. I may be wrong here though

Well I think it is. I made my drawing by reading the source code, and trying to visualize it. The code for this part you can find [here](https://github.com/FlowFuse/node-red-dashboard/blob/main/nodes/store/state.js#L21).

> [@Colin](#):
>
> but at the moment the state store is not available to third party nodes,

Ah damn yes. I had fixed that last week in my local dashboard clone, and I forgot to create a pull request for it. Will do it this evening.

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [9 July 2024 16:51 UTC](https://discourse.nodered.org/t/how-to-sync-the-full-config-between-server-and-client-side-state/89252/26 "2024-07-09T16:51:52Z")

</div>

> [@BartButenaers](#):
>
> Well I think it is. I made my drawing by reading the source code, and trying to visualize it. The code for this part you can find [here](https://github.com/FlowFuse/node-red-dashboard/blob/main/nodes/store/state.js#L21)

Isn't that code for the data store?

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [9 July 2024 17:43 UTC](https://discourse.nodered.org/t/how-to-sync-the-full-config-between-server-and-client-side-state/89252/27 "2024-07-09T17:43:23Z")

</div>

> [@Colin](#):
>
> Isn't that code for the data store?

No, you can find [here](https://github.com/FlowFuse/node-red-dashboard/tree/07f2d2b55e1f85af63cd32f9e11cfc6f14206a85/nodes/store) the code for both stores. But you are right that my drawing is incomplete. Seems that in _ **both** _ stores there is no set/save executed when the message contains a socketId.

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [9 July 2024 18:06 UTC](https://discourse.nodered.org/t/how-to-sync-the-full-config-between-server-and-client-side-state/89252/28 "2024-07-09T18:06:02Z")

</div>

It seems you are right. Though my experience suggests that none of this applies to third party nodes.

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [9 July 2024 18:53 UTC](https://discourse.nodered.org/t/how-to-sync-the-full-config-between-server-and-client-side-state/89252/29 "2024-07-09T18:53:13Z")

</div>

> [@Colin](#):
>
> Though my experience suggests that none of this applies to third party nodes.

That can be the case. But I don't see that at first sight in the source code.

I have updated both stores in the diagram:

- Data store:

- State store:

Here is the updated drawing: [node\_red\_dashboard\_4.txt](https://discourse.nodered.org/uploads/short-url/fvZxSAEIOKhHj0uH9yxOBWOmTEz.txt) (441.5 KB)

> [@BartButenaers](#):
>
> it is unfortunately still not clear to me why we need a _ **datastore** _ to store messages (which are replayed afterwards) while the _ **statestore** _ already does a good job to store the state

For example in the ui\_chart node, the data points are [converted](https://github.com/FlowFuse/node-red-dashboard/blob/main/nodes/widgets/ui_chart.js#L159) to messages and then these messages are appended to the Data store. It is not clear to me why we don't simply add a property "datapoints" to the State store. Then only _an append function needs to be added to the api of the State store_, and you can simply add the datapoint objects to the State store. That way both the server and client side code will become much more simple.

I assume there are cases that I didn't think about yet, and that cannot be solved with the current state store. But I "think" that for most of the cases the state store will be sufficient. And that some other cases might be solved by pimping the state store a bit. For example: _it might perhaps be useful to add - beside "widgetId" and "value" columns - an extra third column "socketId" to the state store. So that data specific to a single client can be recognized that way? That way the get method can decide - based on the socketId - to return only client independent or client specific data._

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [9 July 2024 20:12 UTC](https://discourse.nodered.org/t/how-to-sync-the-full-config-between-server-and-client-side-state/89252/30 "2024-07-09T20:12:44Z")

</div>

> [@BartButenaers](#):
>
> > [@Colin](#):
> >
> > Though my experience suggests that none of this applies to third party nodes.
> 
> That can be the case. But I don't see that at first sight in the source code.
> 
> I have updated both stores in the diagram:

I see that the check on msg.\_client is not where I thought it was so again I may well be wrong. I will run some tests to see what is happening in practice.

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [9 July 2024 20:50 UTC](https://discourse.nodered.org/t/how-to-sync-the-full-config-between-server-and-client-side-state/89252/31 "2024-07-09T20:50:34Z")

</div>

One thing missing from the chart (I think) is that msg.\_client is only relevant if the node type has been configured to pay attention to it in the Client Data tab.  
However, even with that selected, I do not see it stopping the data store being updated. In my gauge node I am using the data store to save the node's state. I do this in the .js file in `onInput` by doing

```auto
 // pick up existing stored data
 let storedData = base.stores.data.get(node.id)
 
 // update storedData from data passed in msg
 ...
 // store the latest full state in our Node-RED datastore
 base.stores.data.save(base, node, storedData)
 // send the message with modified properties to the clients
 send(msg)

```

If I enable the client data setting for the node and then send it data from ui-buttons (so that msg.\_client is present) then the effect is that the `send(msg)` only sends to the gauge in the browser window where the button was clicked. So if I send a new needle value the needle for that connection moves, but it does not move in another browser window. However, if I then refresh the second window the needle moves to the new position, showing that the data was stored in data store even though the client data was present.

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [9 July 2024 21:36 UTC](https://discourse.nodered.org/t/how-to-sync-the-full-config-between-server-and-client-side-state/89252/32 "2024-07-09T21:36:27Z")

</div>

> [@Colin](#):
>
> One thing missing from the chart (I think) is that msg.\_client is only relevant if the node type has been configured to pay attention to it in the Client Data tab.

> [@Colin](#):
>
> One thing missing from the chart (I think) is that msg.\_client is only relevant if the node type has been configured to pay attention to it in the Client Data tab.

Yes true.  
I had not included it in my diagram, because it was already becoming quite complex.  
But indeed it is a rather important piece in the context of this discussion.

- The `msg._client` is being set in the [addConnectionCredentials](https://github.com/FlowFuse/node-red-dashboard/blob/main/nodes/utils/index.js) which is called at different locations.
- Then that value is being checked in both stores, e.g. [here](https://github.com/FlowFuse/node-red-dashboard/blob/main/nodes/store/data.js#L16).

I will add it tomorrow to my diagram, and have a look at your last feedback.  
My time is up for today...

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [9 July 2024 21:47 UTC](https://discourse.nodered.org/t/how-to-sync-the-full-config-between-server-and-client-side-state/89252/33 "2024-07-09T21:47:47Z")

</div>

BTW I got this reminder from Discourse:

![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/7/d/7d6aed0c59614f5e8b12695b6000e45d18731527.png)

So in fact this -seemingly general looking - message contains quite a lot of useful information:

- Discourse has calculated - based on the responses - that the content of our discussion is of no interest at all for the other community members.
- As a result that there is a near zero chance that will ever achieve a positive result with only the two of us.
- As a result we have been labelled as two pathetic old guys, that should stop having useles public discussions about the dashboard design 😋

---

<div class="post-metadata">

**Author:** ![krambriw](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/krambriw/32/5429_2.png) [@krambriw](https://discourse.nodered.org/u/krambriw)\
**Post date:** [10 July 2024 04:53 UTC](https://discourse.nodered.org/t/how-to-sync-the-full-config-between-server-and-client-side-state/89252/34 "2024-07-10T04:53:59Z")

</div>

Oh no, keep up the discussion in public, I do read, it is is really interesting, but I cannot contribute on this very advanced level you both are at 👏

And for the records, and for the core developers I would assume,,,,,this is fundamental requirement for ALL client server systems and there should be a generic functionality in the core so that not every developer has to invent the wheel again

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [10 July 2024 05:57 UTC](https://discourse.nodered.org/t/how-to-sync-the-full-config-between-server-and-client-side-state/89252/35 "2024-07-10T05:57:59Z")

</div>

> [@Colin](#):
>
> showing that the data was stored in data store even though the client data was present.

Morning Colin,  
I need to get off to the daily job, but you can extend your development skills further if you want by debugging the dashboard code to see what is going on. If you want to get start easily you can use simply your Chrome debugger:

1. Install my _ **node-red-contrib-inspect** _ node.

2. Import the [example flow](https://github.com/bartbutenaers/node-red-contrib-inspector?tab=readme-ov-file#node-usage)

3. Hit the Inject button to put your NodeJs instance into debug mode.

4. Open _ **chrome://inspect/#devices** _ in Chrome

5. Make sure the ip address of your device (running Node-RED) is in the configured list with port 9229:

6. After a few seconds an "Inspect" link should appear on your page for that device:

7. In the "Sources" tabsheet you can do "Ctrl-p" to show the available sources (on NodeJs):

8. If you enter data.js you will see the datastore (or state.js for the statestore):

9. Now you can click in the line number bar to add a breakpoint:

10. Probably your browser will immediately stop here already, because you are running your dashboard live. Then you can right click on that breakpoint icon and set a condition:

11. Redo your actions in the dashboard until your arrive again at your breakpoint. Now you can e.g. step through your code:

12. When you arrive at your breakpoint, you can always click on any level in the call stack on the right to see which functions have been executed before to arrive at your breakpoint. And at any level you can see the content of the (local) variables at that point:

Good luck (if you should give it a go!!!!

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [10 July 2024 12:47 UTC](https://discourse.nodered.org/t/how-to-sync-the-full-config-between-server-and-client-side-state/89252/36 "2024-07-10T12:47:48Z")

</div>

> [@BartButenaers](#):
>
> Install my _ **node-red-contrib-inspectd** _ node.

Argh! Why haven't I used this before!!

Well, I am now. 🙂

![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/d/f/df5ebfe1b3d888aa13e68787839ddb07fa0de1dc.png)

Where the function node contains:

```auto
console.log('just testing the debugger again', {msg, node, RED})

debugger

```

Which gives me this in the debug console:

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/4/b/4ba6e4154435c7001c228ecae4e9af983afe7590.png)

Which is also a great way to see the contents of the msg, the node and RED objects.

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [10 July 2024 21:07 UTC](https://discourse.nodered.org/t/how-to-sync-the-full-config-between-server-and-client-side-state/89252/37 "2024-07-10T21:07:12Z")

</div>

> [@BartButenaers](#):
>
> at any level you can see the content of the (local) variables at that point:

Ok, that is one issue solved. It is the old question of the use of the name `msg`. It is not testing whether the incoming message has the property `_client` it is testing whatever is being written to the store. Since the data I am writing to the store is not the incoming message, but is my node state data, it does not contain an `_client` property.  
Whether I should be copying `_client` from the incoming msg into the data to be stored I don't yet know.

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [10 July 2024 21:16 UTC](https://discourse.nodered.org/t/how-to-sync-the-full-config-between-server-and-client-side-state/89252/38 "2024-07-10T21:16:43Z")

</div>

> [@Colin](#):
>
> Ok, that is one issue solved

Hmm, no it isn't. If I trace through that code I can see that it not saving the data, because \_client is present. However, if in my js file after the call to `base.stores.data.save()` I fetch it back from the store I can see that it has in fact saved it (including the \_client property), so something odd is going on. That is a problem for tomorrow though.

Edit: the above is with \_client copied from the message into my node state object.

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [11 July 2024 07:42 UTC](https://discourse.nodered.org/t/how-to-sync-the-full-config-between-server-and-client-side-state/89252/39 "2024-07-11T07:42:09Z")

</div>

> [@Colin](#):
>
> if in my js file after the call to `base.stores.data.save()` I fetch it back from the store I can see that it has in fact saved it (including the \_client property), so something odd is going on

Of, course, I am stupid. I cannot imagine how many times have I explained to forum posters about the fact that javascript accesses data by reference. When I do `let storedData = base.stores.data.get(node.id)` it gives me direct access to the data store object, which I then modify, so when I call `save` I have already updated the object. For this to work I would have to clone the store and then save it again.

In fact I am not going to bother doing this, as now that the state store is available I will change the node to use that, which has get and set property methods which will actually make the code simpler, I think.

---

<div class="post-metadata">

**Author:** ![joepavitt](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/joepavitt/32/59722_2.png) [@joepavitt](https://discourse.nodered.org/u/joepavitt)\
**Post date:** [11 July 2024 08:37 UTC](https://discourse.nodered.org/t/how-to-sync-the-full-config-between-server-and-client-side-state/89252/40 "2024-07-11T08:37:41Z")

</div>

> [@Colin](#):
>
> In fact I am not going to bother doing this, as now that the state store is available I will change the node to use that, which has get and set property methods which will actually make the code simpler, I think.

Just to add, after reading through all this, I am very very tempted to modify the `datastore` APIs to support the approach you'd like.

I prefer the idea of having `state` for anything linked to the Node-RED config (and relevant overrides), and then the `data` store linked to data it renders, e.g. `msg.payload`.

Re-architecting the `datastore` to support both use as a `msg` history/latest store, _and_ a place to store an accumulated picture of the live `data` that a node is using feels very sensible to me?

[Previous page](https://discourse.nodered.org/t/how-to-sync-the-full-config-between-server-and-client-side-state/89252.md?page=1)

[Next page](https://discourse.nodered.org/t/how-to-sync-the-full-config-between-server-and-client-side-state/89252.md?page=3)