I will have a look at your drawing in detail, but I am tied up for a few days so I may not be able to provide feedback promptly.
1 Like
No hurry Colin. I had a look at some dashboard code, which provided me some more insights in how it currently works. Will need to update my diagram with all this information, so we can continue this discussion.
Ok, I am back from the drawing table.
The fog has a little bit cleared in my head.
Although I have limited my drawing to only the data streams in the dashboard, unfortunately it contains much more arrows as I had hoped. But I really need to see everything that is going on behind the scenes, in order to understand how it works. And all those arrows are in fact the reason why I had to draw it, in order to support my brain to understand how it works...
Here is a (large) printscreen:
And here is the corresponding drawing file, in case anybody wants to open it (in https://excalidraw.com/). Note that I had to change the extension from .excalidraw to .txt because it was rejected by Discourse: node_red_dashboard_3.txt (413.9 KB)
My questions and proposals are all in red. I would appreciate if somebody could take the time to review it, in order to make this a useful diagram for UI node developers and dashboard contributors.
When looking at this diagram, it is unfortunately still not clear to me why we need a datastore to store messages (which are replayed afterwards) while the statestore already does a good job to store the state. The datastores and related message replaying makes the development pretty complex to understand.
Hopefully my feedback isn't interpreted as negative, because dashboard D2 has become a piece of art in a small amount of time. And I am still a big fan of it! But imho the message-based approach does really increase the threshold to start contributing.
Thanks for reading!!!
Bart
Only to the newly connected client.
I am still confused about some of this though. As I understand it, on connection, the complete state of all widgets is sent in the ui-config object, and if the developer tools is enabled this can be seen. However, for third party nodes at least, this is not made directly available. What is made available is the mounted function which is given this.props which is the node's config merged with the datastore.
i don't think there is any check on check on msg._client for the update of the state store. I may be wrong here though.
That is as far as I have got at the moment.
Ditto. In any new nodes I am working on I am, for the moment, ignoring the data store, and not writing incoming messages to it, so I do not get the last message replayed on connection. Everything I need is in the state store.
[Edit] At least I would do, but at the moment the state store is not available to third party nodes, so I am having to use just the data store instead, but using it as a state store, not saving messages to it.
@Colin,
Thanks for reviewing! Really appreciated!!
Well I think it is. I made my drawing by reading the source code, and trying to visualize it. The code for this part you can find here.
Ah damn yes. I had fixed that last week in my local dashboard clone, and I forgot to create a pull request for it. Will do it this evening.
Isn't that code for the data store?
No, you can find here the code for both stores. But you are right that my drawing is incomplete. Seems that in both stores there is no set/save executed when the message contains a socketId.
It seems you are right. Though my experience suggests that none of this applies to third party nodes.
That can be the case. But I don't see that at first sight in the source code.
I have updated both stores in the diagram:
-
Data store:
-
State store:
Here is the updated drawing: node_red_dashboard_4.txt (441.5 KB)
For example in the ui_chart node, the data points are converted to messages and then these messages are appended to the Data store. It is not clear to me why we don't simply add a property "datapoints" to the State store. Then only an append function needs to be added to the api of the State store, and you can simply add the datapoint objects to the State store. That way both the server and client side code will become much more simple.
I assume there are cases that I didn't think about yet, and that cannot be solved with the current state store. But I "think" that for most of the cases the state store will be sufficient. And that some other cases might be solved by pimping the state store a bit. For example: it might perhaps be useful to add - beside "widgetId" and "value" columns - an extra third column "socketId" to the state store. So that data specific to a single client can be recognized that way? That way the get method can decide - based on the socketId - to return only client independent or client specific data.
I see that the check on msg._client is not where I thought it was so again I may well be wrong. I will run some tests to see what is happening in practice.
1 Like
One thing missing from the chart (I think) is that msg._client is only relevant if the node type has been configured to pay attention to it in the Client Data tab.
However, even with that selected, I do not see it stopping the data store being updated. In my gauge node I am using the data store to save the node's state. I do this in the .js file in onInput by doing
// pick up existing stored data
let storedData = base.stores.data.get(node.id)
// update storedData from data passed in msg
...
// store the latest full state in our Node-RED datastore
base.stores.data.save(base, node, storedData)
// send the message with modified properties to the clients
send(msg)
If I enable the client data setting for the node and then send it data from ui-buttons (so that msg._client is present) then the effect is that the send(msg) only sends to the gauge in the browser window where the button was clicked. So if I send a new needle value the needle for that connection moves, but it does not move in another browser window. However, if I then refresh the second window the needle moves to the new position, showing that the data was stored in data store even though the client data was present.
Yes true.
I had not included it in my diagram, because it was already becoming quite complex.
But indeed it is a rather important piece in the context of this discussion.
- The
msg._clientis being set in the addConnectionCredentials which is called at different locations. - Then that value is being checked in both stores, e.g. here.
I will add it tomorrow to my diagram, and have a look at your last feedback.
My time is up for today...
BTW I got this reminder from Discourse:
So in fact this -seemingly general looking - message contains quite a lot of useful information:
- Discourse has calculated - based on the responses - that the content of our discussion is of no interest at all for the other community members.
- As a result that there is a near zero chance that will ever achieve a positive result with only the two of us.
- As a result we have been labelled as two pathetic old guys, that should stop having useles public discussions about the dashboard design
5 Likes
Oh no, keep up the discussion in public, I do read, it is is really interesting, but I cannot contribute on this very advanced level you both are at 
And for the records, and for the core developers I would assume,,,,,this is fundamental requirement for ALL client server systems and there should be a generic functionality in the core so that not every developer has to invent the wheel again
6 Likes
Morning Colin,
I need to get off to the daily job, but you can extend your development skills further if you want by debugging the dashboard code to see what is going on. If you want to get start easily you can use simply your Chrome debugger:
-
Install my node-red-contrib-inspect node.
-
Import the example flow
-
Hit the Inject button to put your NodeJs instance into debug mode.
-
Open chrome://inspect/#devices in Chrome
-
Make sure the ip address of your device (running Node-RED) is in the configured list with port 9229:
-
After a few seconds an "Inspect" link should appear on your page for that device:
-
In the "Sources" tabsheet you can do "Ctrl-p" to show the available sources (on NodeJs):
-
If you enter data.js you will see the datastore (or state.js for the statestore):
-
Now you can click in the line number bar to add a breakpoint:
-
Probably your browser will immediately stop here already, because you are running your dashboard live. Then you can right click on that breakpoint icon and set a condition:
The debugger will only stop here then if the (javascript) condition is true, so you can e.g. check for your node id.
-
Redo your actions in the dashboard until your arrive again at your breakpoint. Now you can e.g. step through your code:
-
When you arrive at your breakpoint, you can always click on any level in the call stack on the right to see which functions have been executed before to arrive at your breakpoint. And at any level you can see the content of the (local) variables at that point:
Good luck (if you should give it a go!!!!
3 Likes
Argh! Why haven't I used this before!!
Well, I am now. 
Where the function node contains:
console.log('just testing the debugger again', {msg, node, RED})
debugger
Which gives me this in the debug console:
Which is also a great way to see the contents of the msg, the node and RED objects.
2 Likes
Ok, that is one issue solved. It is the old question of the use of the name msg. It is not testing whether the incoming message has the property _client it is testing whatever is being written to the store. Since the data I am writing to the store is not the incoming message, but is my node state data, it does not contain an _client property.
Whether I should be copying _client from the incoming msg into the data to be stored I don't yet know.
Hmm, no it isn't. If I trace through that code I can see that it not saving the data, because _client is present. However, if in my js file after the call to base.stores.data.save() I fetch it back from the store I can see that it has in fact saved it (including the _client property), so something odd is going on. That is a problem for tomorrow though.
Edit: the above is with _client copied from the message into my node state object.
Of, course, I am stupid. I cannot imagine how many times have I explained to forum posters about the fact that javascript accesses data by reference. When I do let storedData = base.stores.data.get(node.id) it gives me direct access to the data store object, which I then modify, so when I call save I have already updated the object. For this to work I would have to clone the store and then save it again.
In fact I am not going to bother doing this, as now that the state store is available I will change the node to use that, which has get and set property methods which will actually make the code simpler, I think.
1 Like
Just to add, after reading through all this, I am very very tempted to modify the datastore APIs to support the approach you'd like.
I prefer the idea of having state for anything linked to the Node-RED config (and relevant overrides), and then the data store linked to data it renders, e.g. msg.payload.
Re-architecting the datastore to support both use as a msg history/latest store, and a place to store an accumulated picture of the live data that a node is using feels very sensible to me?
