Uibuilder: to subscribe or not to subscribe?

[tve]

I'm trying to move on from the std dashboard: not easy... (It has become too limiting and too slow for me.) One of the things I'm trying to wrap my mind around is whether to introduce subscriptions into the dashboard model.

One of the complexities of adding a dashboard to node-red is how to connect the nodes in node-red with components in the web browser. For that some identifiers are needed. In the std node-red dashboard these identifiers are hidden inside the UI nodes, which is nice for the user, but ultimately limiting. With uibuilder there is no built-in matching, so in principle anything is possible, but looking at uibuilder-vuejs-component-extras the identifiers are made explicit: the user gives each component a "ref ID" and then targets a component by setting that ref ID on messages sent to the uibuilder node. Cinhcet's component-dashboard does something similar by matching messages on the HTML element ID.

Once we have these routing identifiers to match producer and consumer of messages we can start asking how exactly they are matched and when messages are forwarded and what else can be done to messages (such as caching/retaining). For example, in most uses uibuilder needs a caching mechanism to initialize components when a new client connects.

In uibuilder with vuejs-components-extra and in component-dashboard the producer/consumer matching is done on full ID equality. But looking at pub/sub, such as MQTT, generally some form of ID (topic) hierarchy is desirable and supported by some form of prefix matching. So is a topic hierarchy useful in a dashboard implementation as well?

In uibuilder and component-dashboard the producer controls message forwarding. It seems that in general all messages are sent to all clients. Uibuilder adds a SocketID which lets the producer control message forwarding to a single client. In contrast, pub/sub systems let the consumer control message forwarding using subscriptions. Which of the two models is more appropriate for a dashboard implementation?

I believe that the general trend in web design has been to let the consumer control data forwarding: the components on each web page request data from the server and, where a live feed is desired, subscribe to such a feed. In my experience the reason for this trend is that it reduces the coupling between front-end and back-end code: the back-end (server) provides RESTful access to its data in a way somewhat agnostic to the details of the web apps, and the front-end (web components) pull the data they currently need.

How message forwarding is controlled also has performance and complexity aspects. On the performance side a subscription model can ensure that only the data actually needed by the client is transmitted. In node-red dashboard terms,this means that if a single-page-app has 20 tabs (like my current dashboard...) it can choose to only subscribe to the topics relevant to the components displayed on the current tab. On the downside, the subscription model requires subscription messages to be sent and it requires that the server track subscriptions for each client, which adds quite some complexity especially if prefix matching is supported.

I've been using quite a few MQTT terms so far because my node-red implementation does almost all its I/O through MQTT, at least all the devices are attached through MQTT. This leads me to wonder what would happen if the dashboard were attached to node-red via MQTT? Consider the following variation on uibuilder:

• instead of sending messages to the uibuilder node just send to the MQTT broker, control caching by setting the retain flag
• instead of connecting to the node-red process, the dashboard connects directly to the broker via a websocket, speaks MQTT, and subscribes to the topics it needs for display

I believe this would accomplish a number of things:

• supports pub/sub so web pages can subscribe to just the data they need
• supports "caching" through the MQTT retain facility, so a new client gets the last message sent to the topics it subscribes to
• supports direct interaction between IoT devices and the UI: one doesn't have to create a node-red flow just to funnel messages through (this also raises permissions/security issues)
• supports a more general architecture where node-red is just one service on the "MQTT bus" and the dashboard can just as easily interact with additional services via MQTT

Of course attaching the dashboard to MQTT is not attractive for node-red deployments that don't use MQTT and it is also true that a pub/sub mechanism as well as caching could be implemented in node-red/uibuilder/component-dashboard. And there are cons to using MQTT:

• the retain flag only controls retention of the last message, when drawing charts it can be useful to retain the last N messages or the messages for some time-period, which is something a custom caching mechanism around uibuilder could implement
• permissions need to be handled at the MQTT level, which may or may not be a good match for the task
• uibuilder implements quite some machinery to serve-up the UI, including the ability to edit the UI from within node-red

I must say that after thinking through all this I find the pub/sub model for the dashboard very compelling. I also find an MQTT-attached dashboard conceptually very compelling but I'm not sure about the practical aspects (permissions and serving-up the UI). I'd be very interested to hear from others that have much more experience than I do in implementing a dashboard for node-red!

[TotallyInformation]

@tve

One of the things I'm trying to wrap my mind around is whether to introduce subscriptions into the dashboard model.

Phew, big questions here!! I'll do my best to go through them but do shout out if I miss something.

@tve

With uibuilder there is no built-in matching, so in principle anything is possible, but looking at uibuilder-vuejs-component-extras the identifiers are made explicit: the user gives each component a "ref ID" and then targets a component by setting that ref ID on messages sent to the uibuilder node. Cinhcet's component-dashboard does something similar by matching messages on the HTML element ID.

Yes, that is, of course, a deliberate choice. You should be using the element ID to match data to elements where you can. But sometimes you may need to match to at CLASS instead of an ID.

Unfortunately, VueJS components do not pass the element ID to the component but instead have to implement their own REF attribute. Annoying but there is probably no way around that, HTML wasn't designed with these things in mind.

So that is 3 mechanisms already. This is why uibuilder bravely (:cough:) avoids the question mostly and leaves it up to the author/designer who likely know best what will work for their app.

The VueJS helpers are only a partial solution and really only designed to help reduce coding for people who wish to avoid it. It cannot be a full solution because it still depends on the Vue component author doing things in a certain way. However, you can generally work round that if you are prepared to write your own components that wrap others. That's what I did for the component-extras library so that at least people have an idea of what is possible. Obviously I barely have time to work on uibuilder let alone write loads of component wrappers. K.I.S.S.

Further, because uibuilder is not limited to VueJS, those helpers will never take over the library, the whole point of uibuilder is to allow you to use any (or no) front-end framework. There is no generic data-to-UI model. Though I'm always open to ideas and suggestions

@tve

Once we have these routing identifiers to match producer and consumer of messages we can start asking how exactly they are matched and when messages are forwarded and what else can be done to messages (such as caching/retaining).

We can indeed start - but can we agree? If not, perhaps there is room for some "middleware" that acts as a data router and is pluggable so that different data-routers can be plugged in?

@tve

For example, in most uses uibuilder needs a caching mechanism to initialize components when a new client connects.

Not uncommon. Dashboard includes something. But again, when I looked at it in detail, I realise that there are so many different caching models, there was no way to come up with something totally generic without building something enormous which wouldn't be feasible to either use or maintain. Though one day, if time ever allows, I'll pick up the caching nodes again, there is a package for it but it is a long way down the priority order - I'll happily take collaborators though.

@tve

So is a topic hierarchy useful in a dashboard implementation as well?

It would be complex. You would need to come up with some concrete use-cases I think before anyone would want to start working on it. If you came up with a model (or maybe several), we could again look at whether we could find a way to plug them in as standard extensions to the uibuilderfe library. That would probably make most sense because I can't ever see everyone agreeing on a single data-routing model that covers all scenarios.

@tve

It seems that in general all messages are sent to all clients.

It only seems that way. Again, this is because you need a way to identify each client - Remembering that a specific tab in a specific browser on a specific device defines a client, a single device may result in many clients - should they have the same id or different ids? The answer, of course, is "it depends". And therein lies the rub. Too many possibilities. Either we have to choose one and therefore lock out other possibilities or try to cater for many and tie ourselves up in coding for the next 10 years.

Or, as I'm slowly trying to do, we come up with a simple model - e.g. standardised data model - that lets the author write their own code to do whatever they need to. That is the model I am implementing in the "security model" that was meant to go live with v3 but ended up taking too long to finalise and so is currently partially implemented.

@tve

In contrast, pub/sub systems let the consumer control message forwarding using subscriptions

This is something that I would certainly like to look at further. I think that something could be done in Socket.IO to allow this but I've not got my head around what it would actually look like. I'm certainly open in this area if you can think of some ways we could start to look at it.

@tve

This leads me to wonder what would happen if the dashboard were attached to node-red via MQTT?

The problem here is that it is another protocol and realistically it requires a separate broker. I know that there is a broker node but I really don't see that as a feasible, scalable or securable option.

@tve

control caching by setting the retain flag

Just a side-note here that this is not a full caching option. It only allows 1 msg per topic to be retained. That is not sufficient for all models of caching.

@tve

instead of connecting to the node-red process, the dashboard connects directly to the broker via a websocket, speaks MQTT, and subscribes to the topics it needs for display

This is, of course doable without any changes to uibuilder. All you need is an MQTT library added to your front-end code. Then just don't send anything to uibuilder Job done.

Would it really be worth the added complexity of adding this to uibuilderfe? Probably not. You could already wire incoming MQTT msg's to uibuilder's msg input probably (I've not tested that but uibuilder includes a setter function and all it would need would be a minor change to allow you to update the library's msg variable - I think I blocked it but it is only 1 line of code). Then by manually setting the msg object, you could link incoming MQTT messages to the uibuilder msg so that the same onChange function handles both types of input.

@tve

uibuilder implements quite some machinery to serve-up the UI, including the ability to edit the UI from within node-red

This really isn't an issue here. You could easily add the front-end MQTT library to uibuilder's list of served modules. The editor is just that, it helps you edit your front-end code, nothing to see here, it just works. And you don't have to use it, it is just a convenience.

The last release of uibuilder also added a more flexible templating option that is set to continue to improve. At the moment, you can only choose between the templates supplied. However, I intend to implement a feature that will let you use templates direct from GitHub - so anyone could create an MQTT-flavoured template to make it easy for people to get started.

@tve

the pub/sub model

As I say, I would most certainly be interested in hearing from anyone who has ideas about how this could be done using Socket.IO. I think that would be a nice extension to uibuilder and it still leaves control in the hands of the designer/author where it belongs (not in the hands of the framework author - e.g. me!).

The MQTT connection is only a solution in certain use-cases and isn't fully generic. So I wouldn't be building that into uibuilder. However, I could certainly be persuaded to allow authors to use a setter to "pretend" to send a msg from Node-RED but all in the front-end which would allow you to have a single set of UI update code with multiple data-sources. I would also be more than happy for someone to come up with some template code that I could add as another example template to the next release that shows how to do this.

[tve]

Thanks for all the thoughts!

I must say I really like the idea of subscriptions. It decouples the producer and consumer better, I think. Also, I have a pretty clear producer namespace 'cause all the devices and metrics have names/ids, so subscribing from that is quite clear and stable. The UI is more fluid with stuff being moved around, grouped differently, etc. I'll see how it looks in practice...

I'm less convinced about going to MQTT directly, however. Being able to customize caching/retention and having a better grip on permissions seems worth going through node-red. If I use subscriptions I can always add a way for a UI component to directly subscribe to an MQTT topic.

[TotallyInformation]

Is there a next step then?

[tve]

The next step for me is to try subscriptions out by hacking something up.

[chrisn-au]

Guys,

I am very new to this thread /problem really only having the need to explore alternate dashboards yesterday. My starting feelings

1. I really really like the near zero config low barrier for entry main dashboard - it is amazing
2. I absolutely agree with @BartButenaers comments in other threads that ideally we have a unified approach to dashboards otherwise support from the open source community will not just be there. I would love to reuse / extend the capability of existing capability
3. There may be others but UIBuilder from @TotallyInformation seems like a very sensible platform independent comms mechanism between a dashboard and node-red. It seems very silly to develop an alternate mechanism

Where the current dashboard breaks down for me is the lack of multi user capability. It is an amazing bit of code but simply does not work in a multi user environment. From my needs from an end user POV it is so close but lack of multi user means that it just can work for a number of my use cases.

I mainly use flutter for SPA and any cross platform app development. I am a backend guy but learn enough front end technologies to get by. The reason I like flutter is the declarative UI. It really works for me. I know there are lots of other technologies but flutter works for me.

I have been exploring the Admin API (for a seperate documentation project. I have way too many instances and tabs to keep it all in my head. I am looking at generating documentation form the instances ) and realised that it gives all the information that a declarative UI needs

type
group
positioning
type specific data

and that at a very basic level all that I really need to do is to map node red dashboard widgets to flutter UI widgets and with a little bit of magic and it should just work. Multi user and caching problems aside (and I accept they are potentially non-trivial, but seem to be able to be defined in current widgets even if not implemented).

This in theory would allow for an alternate dashboard keeping definition in Node Red giving hints to the dashobard implementor on content and layout. It is up to that implentation whether it takes the hints or ignores and allows for totally custom dev.

As a sunday afternoon project I am going to see if I can get a POC using very simple widgets to see if it makes sense

Does this make sense ?

[TotallyInformation]

Your thinking is, I suspect quite close to my own in regard to uibuilder. Except that you are focused on Flutter rather than VueJS. This is understandable given your direction of travel. And it highlights some of the core reasoning behind uibuilder. The need to accommodate so very many different front-end libraries and approaches.

With uibuilder, my thinking has been to develop the core node and front-end library to be as flexible but easy to use as possible. My limited time has meant that, so far, I've not had time to dedicate to the 2nd phase which is to build standardised components that would, as you say, map to similar Dashboard components.

What is clear to me however, is that I don't think we will ever end up with a single collection of components because they are always going to be tied to a specific front-end framework (just as Dashboard is).

What I really want to do therefore, is come up with a SPECIFICATION for how the components should interact with Node-RED. With that in hand, anyone could create components in whatever framework they like. If anyone would like to collaborate on that effort, I'd be delighted.

But there is another part to Dashboard that I've not even begun to investigate - mainly because it is beyond my current knowledge levels. That is a page layout designer. Dashboard has the components and a page layout designer and both are, in my view, required for anything to be considered a possible replacement. Maybe that doesn't need to be a fully visual, general purpose layout designer - Dashboard isn't for example since it constrains you to tabs, groups and cards mostly - though it would, of course, be nice.

In regard to Flutter, which I've not come across before, I do note that for building a web app, it does indeed produce an index.html file and therefore will certainly be compatible with uibuilder. I've not tested it but as we saw with Svelte, the integration was really easy. Whether or not the dev server will be as uibuilder/node-red friendly as the Svelte one was is hard to predict though - if you get a chance to test it, that would be great.

It would also be very interesting to see a mobile flutter app with uibuilder, that might be a bit more challenging but I'd be happy to help if you get stuck The main initial complexity would be getting the uibuilder.start() parameters correct and that really shouldn't be very hard. Then security would be the next complexity if you want the app to work over the Internet. While the built-in security features of uibuilder are not quite 100% ready yet, there still should be more than enough flexibility to put something together.

@chrisn-au

As a sunday afternoon project I am going to see if I can get a POC using very simple widgets to see if it makes sense

Does this make sense ?

Yes, but again, we need a collective effort on standardising the INTERFACE. What data SCHEMA are we going to define? If you look into uibuilder's tech docs (which you can get to from a uibuilder node or from GitHub), you will see that I've started to look at standardising some msg structures and I believe this is key to any future Dashboard replacement. Having a standard interface will mean that we can have multiple dashboard implementations to meet specific needs without everyone having to learn different ways to interface with them, in theory they would be a plug-and-play replacement.

[chrisn-au]

@TotallyInformation

The need to accommodate so very many different front-end libraries and approaches.

Agreed

@TotallyInformation

What is clear to me however, is that I don't think we will ever end up with a single collection of components because they are always going to be tied to a specific front-end framework (just as Dashboard is)

Agreed although I feel any serious FE implementstion would need to commit to a certain set of components. In flutter speak there are a number of plugins that will only work with ios/android and will not work with web

@TotallyInformation

What I really want to do therefore, is come up with a SPECIFICATION for how the components should interact with Node-RED. With that in hand, anyone could create components in whatever framework they like. If anyone would like to collaborate on that effort, I'd be delighted .

That I absolutely concur with. I would love to see a uibuilder interface spec that documents the contracts between ui builder node red and that FE interfsce. It is a bit of work but clesrly it is essential to any serious implementstion. I am happy to help.

@TotallyInformation

But there is another part to Dashboard that I've not even begun to investigate - mainly because it is beyond my current knowledge levels

I am pretty close probably a couple of days to having a proof of concept of flutter UI generation based on the existing dashboard

[{"id":"2397158.10a1eea","type":"debug","z":"88c95266.1d416","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":710,"y":450,"wires":[]},{"id":"b1e0800a.4f00a","type":"inject","z":"88c95266.1d416","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":175,"y":450,"wires":[["db412636.b72a6"]]},{"id":"db412636.b72a6","type":"http request","z":"88c95266.1d416","name":"flows","method":"GET","ret":"obj","paytoqs":"ignore","url":"http://localhost:1880/flows","tls":"","persist":false,"proxy":"","authType":"","x":340,"y":450,"wires":[["70db2376.e8c854"]]},{"id":"70db2376.e8c854","type":"function","z":"88c95266.1d416","name":"Filter and reorg UI","func":"let flows = msg.payload\nlet tabs = flows.filter(elm => elm.type == \"ui_tab\" )\nlet uigroups = flows.filter(elm => elm.type.substring(0,8) == \"ui_group\" )\n\nlet uiTabs = []\n\nfor (let i =0; i < tabs.length; i++){\n    tab = tabs[i]\n    let elms = uigroups.filter(elm => elm.tab == tab.id)\n    elms = elms.sort((a, b) => a.order - b.order)\n    console.log(`elms ${elms.length}`)\n    let groups = []\n   \n    for (let j = 0; j < elms.length; j++){\n        group = elms[j];\n        let groupElms = flows.filter(elm => elm.group == group.id)\n        groupElms = groupElms.sort((a, b) => a.order - b.order)\n        groups.push({id : group.id, details : group , element : groupElms})\n    }\n    uiTabs.push({id : tab.id, details: tab, groups : groups})\n}\n\nmsg.payload = uiTabs\n\n\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":510,"y":450,"wires":[["2397158.10a1eea"]]}]

The flow above extracts the UI stuff from the flows, and in the short term I will code generate (in the very short term, once I have a WS connection I will generste on fly) the Flutter UI based on it. I truly love the wyswig aspect of the current dashboard and am very keen to keep it as a part of the soln. Whether there is a more genric designer is a good question for which I am neutral on. My short term preference is to provide an out of the box experience as much as possible. My gut says it is possible to sensibly (effort wise) effectively replicate the current dashboard capability. If i can do tthat for flutter (which would mean that a user could have NO flutter experience), then it would be very good to have an extendable platform to do more fancy non NR dashobard stuff. I know that this is a differnet slant than yours which is great flexibility and I get that, i am sure the answer lies in the middle

For flutter integration, technically the app on start (web or native) will connect to the uibuilder nr (via socket.io) component and request layout. From there it will generate the layout in the app. In theory with this archtecture you will get native app for nothing. It will just work

For the app security, lots to think about but reasonably solved problems. Mandating reverse proxies is ont realistic but that RP and oauth2 login to trusted providers like google, github , etc would be good but down the track.

Happy to help with the documentation. If there is some interest we could arrange a google meet over the next few weeks.

[TotallyInformation]

All looks cool, can't wait to see your results.

@chrisn-au

any serious FE implementstion would need to commit to a certain set of components

Yes, indeed. I would expect to at least see all of the Dashboard core components. Card, text, various inputs, etc. Charts maybe. Templates - should be easily achieved through the standard uibuilder stuff I would think.

One key difference for me is that any future dashboard should not need to send any CODE to the front-end, only data should be sent (can be configuration data or display data). Not to say that it can't send any, uibuilder already allows that, but that it should not require it.

@chrisn-au

I would love to see a uibuilder interface spec that documents the contracts between ui builder node red and that FE interfsce. It is a bit of work but clesrly it is essential to any serious implementstion. I am happy to help.

Great, lets start a separate conversation on this then? Either here on the forum or on the uibuilder repo discussion area? What's your preference?

@chrisn-au

I know that this is a differnet slant than yours which is great flexibility and I get that, i am sure the answer lies in the middle

I realise that any dashboard style overlay that uses uibuilder will need to be specific and that is certainly not an issue. That is, indeed, why I'm determined to keep uibuilder itself as generic as possible. Some VueJS specific things have crept into uibuilderfe recently but that is partly because i'm too lazy I don't have the time to separate out the logic to a separate library while retaining integration. They are really quite experimental features that I'd like to build on so that they become features available to multiple front-end frameworks. Think of them as the starting point for some of the work on the standardised schema's.

@chrisn-au

technically the app on start (web or native) will connect to the uibuilder nr (via socket.io) component and request layout

Wouldn't it be better for your flow to send that on a new connection? That's what the uibuilder control outputs are for.

I'm also open to suggestions for enhancements to uibuilder to make this process easier.

@chrisn-au

For the app security, lots to think about but reasonably solved problems. Mandating reverse proxies is ont realistic but that RP and oauth2 login to trusted providers like google, github , etc would be good but down the track.

I agree, that is a future set of enhancements, you can certainly work around any current limitations as you say.

@chrisn-au

If there is some interest we could arrange a google meet over the next few weeks.

Happy to try and do that. Though I spend my life on conference calls already

Are we in the same timezone though? I'm in the UK. Oh, and to reiterate, this isn't my day-job so I have to squeeze it in around work and family

[chrisn-au]

Hi,

I will create a separate thread for the discussion

Chris

[chrisn-au]

On seconds thoughts -given the amount of content already in this thread, will keep this open. Will create new threads for side topics. @tve apologies for hi jacking your thread, but the discussion is absolutely consistent with your thread. Let me know if you want us to stop using this

In the place where I do all my best thinking, I just realised the while creating a new dashboard from existing flows is I am hoping a sensible problem to solve, I just realised without changes to the UI nodes we will not be able to provide a seamless experience to user.

A mechanism to do this, and I don't know the UI nodes at all, would be for the UI nodes to have an option to use UIBuilder (as the primary or secondary UI). This would require a change to the UI nodes to provide a mechanism to optionally add a transparent pass through to the UIBuilder node. So the text node for example would have an option in config to provide info to UI Builder. This would enable a output on the text node which could be connected to UIBuilder. Alternatively there could be s UI Builder server node config which could be configured on the UI Nodes that would handle the communication.

@knolleary, while this is still obviously in early thought, is a PR to the UI nodes to add an additional path a sensible approach ? If not then the concept of using existing dashboard infrastructure to extend UI Builder falls a bit flat

[chrisn-au]

@dceejay Apologies, I understand you maintain the dashboard nodes. @knolleary sorry for the spam

[TotallyInformation]

@chrisn-au

A mechanism to do this, and I don't know the UI nodes at all, would be for the UI nodes to have an option to use UIBuilder (as the primary or secondary UI). This would require a change to the UI nodes to provide a mechanism to optionally add a transparent pass through to the UIBuilder node

The problem here is that I think that Dashboard works in a totally different way to uibuilder. AFAIK, Dashboard nodes deliver the code to the front-end whereas uibuilder uses standard files for its components.

We really don't want to make Node-RED compose and send code over websockets and then send data, that isn't terribly efficient and isn't very scalable or cacheable. The components should be parameterised bits of display code that only need standard data sending to them and nothing more. The display code can be in files and delivered using standard web server methods.

Check out the example VueJS components in this repo for an example of what I mean. These have been designed so that you can simply send pre-formatted messages from Node-RED to make use of them, all you need to do in the front end is include the library and add a blank element to your HTML.

GitHub - TotallyInformation/uibuilder-vuejs-component-extras: A set of extra...

[chrisn-au]

We are in agreement. I absolutely agree that we do not want to send code via ui builder. My point and probably poorly made, is a change to the UI nodes to transparently pass any data it receives to UIBuilder as well as doing its current dashboard functions (sending code to the dashboard)

This would enable the two dashboards to run in parallel

Chris

[chrisn-au]

Had a fun night last night (clearly I am not a movie maker nd have no intention to be)

A fun experiment withe node red dashboard and flutter

Created a trivial flow which would extract the dashboard components via the admin api (limited to 1 tab, 1 group and ui_text and ui_button) , does some basic source code generation for dart ,saves the file to the flutter library and then reload using flutters awesome hot reload feature

The flutter UI clearly needs work and this is nothing more than a very first step to understand what sort of work flutter UI creation from a dashboard config would be. My view is while there Is non trivial work it is feasible especially for the core nodes.

My next step in proving the technology will be to hook it up to a naive WS connection to allow 2 way comms between the flutter app and node-red.

[TotallyInformation]

Interesting indeed.

Have you tried the integration between Flutter and uibuilder? If that works OK then you may be able to remove the manual steps and get the ws connection via uibuilder.

[chrisn-au]

Hi,

so a little bit of progress. Have got a WS connection between NR and the Flutter app. It will refresh the flutter app on deploy. The next step over the weekend (family permitting) will be to get the data flow going (currently only control flow for sending updated flows) and adhoc data back from button presses. The next step is a bit of SMOP (simple matter of programming) so am not expecting much hassle.

BTW the cool thing about flutter is getting the native app for nothing. I will try and release the flutter app to GitHub over the next week or so.

The flow that does the extract from Node Red

[{"id":"d2ad6df4.275668","type":"http request","z":"6874eab9.ded86c","name":"flows","method":"GET","ret":"obj","paytoqs":"ignore","url":"http://localhost:1880/flows","tls":"","persist":false,"proxy":"","authType":"","x":550,"y":180,"wires":[["94635674.322ce"]]},{"id":"94635674.322ce","type":"function","z":"6874eab9.ded86c","name":"Filter and reorg UI","func":"let flows = msg.payload\nlet tabs = flows.filter(elm => elm.type == \"ui_tab\" )\nlet uigroups = flows.filter(elm => elm.type.substring(0,8) == \"ui_group\" )\n\nlet uiTabs = []\n\nfor (let i =0; i < tabs.length; i++){\n    tab = tabs[i]\n    let elms = uigroups.filter(elm => elm.tab == tab.id)\n    elms = elms.sort((a, b) => a.order - b.order)\n    console.log(`elms ${elms.length}`)\n    let groups = []\n   \n    for (let j = 0; j < elms.length; j++){\n        group = elms[j];\n        let groupElms = flows.filter(elm => elm.group == group.id)\n        let cleansedElms = []\n      //  cleansedElms.push(groupElms);\n        for (let k =0; k < groupElms.length; k++){\n            elm = groupElms[k]\n            delete elm.wires\n            if (elm.type ==  \"ui_button\" || elm.type == \"ui_text\") cleansedElms.push(elm)\n        }\n        cleansedElms = cleansedElms.sort((a, b) => a.order - b.order)\n        groups.push({id : group.id, group_details : group , element : cleansedElms})\n    }\n    uiTabs.push({id : tab.id, tab_details: tab, groups : groups})\n}\n\n\nlet control = {\n    type : \"control\",\n    scope : \"tab_build\",\n    tabs : uiTabs\n    \n} \nmsg.payload = control\n\nmsg.control = control\n\n\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":730,"y":180,"wires":[["86a2cebe.401fe8","a5bf0d6d.6cefe"]]},{"id":"9983e29b.6844e8","type":"websocket out","z":"6874eab9.ded86c","name":"","server":"706934a9.4651ac","client":"","x":740,"y":240,"wires":[]},{"id":"86a2cebe.401fe8","type":"json","z":"6874eab9.ded86c","name":"","property":"payload","action":"","pretty":true,"x":570,"y":240,"wires":[["9983e29b.6844e8"]]},{"id":"fdaf4382.019728","type":"inject","z":"6874eab9.ded86c","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":true,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":390,"y":240,"wires":[["d2ad6df4.275668"]]},{"id":"12dfd1d3.616c2e","type":"status","z":"6874eab9.ded86c","name":"","scope":["8c963eb6.05be18","29b60332.e886a4"],"x":180,"y":180,"wires":[["588e1b39.ca3d24"]]},{"id":"588e1b39.ca3d24","type":"change","z":"6874eab9.ded86c","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"{\"type\":\"Control\",\"scope\":\"flows\"}","tot":"json"}],"action":"","property":"","from":"","to":"","reg":false,"x":380,"y":180,"wires":[["d2ad6df4.275668"]]},{"id":"a5bf0d6d.6cefe","type":"debug","z":"6874eab9.ded86c","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":680,"y":120,"wires":[]},{"id":"1fb5cdef.38c5f2","type":"websocket in","z":"6874eab9.ded86c","name":"","server":"706934a9.4651ac","client":"","x":240,"y":120,"wires":[["d30b5acb.34de58"]]},{"id":"d30b5acb.34de58","type":"debug","z":"6874eab9.ded86c","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":450,"y":120,"wires":[]},{"id":"706934a9.4651ac","type":"websocket-listener","z":"6874eab9.ded86c","path":"/ws/simple","wholemsg":"false"}]

Node Red Flutter Dashboard 2

[chrisn-au]

So have very nearly got all I need for proof of technology for a flutter drop in for an alternative for the dashboard (particularly core nodes)

The demo can

1. Generate UI from flow data on deploy
2. Receive data from Node-Red,
3. Send data back to Node-red.
4. Crude re use of existing UI nodes for definition and data in
5. Run in web, iOS, macOS (yep it just worked) android have not tested android but should just work also. When I publish to Github later next week, I suggest someone builds a window version. I don't have a Linux desktop so won't build for that.

It all seems pretty sensible and from what I can see no reason why it can not be stable. I am not suggesting that this a replacement path for the dashboard but just a concept of generatable UI from the existing UIs.

Before I publish the code I will add support for data-intensive widgets like graphs, charts etc. I cant see a problem with it it is just slightly tricky work to get the architecture right. I think I would like to put some side caching in place for that but will think it through a bit before I hack it together

Regardless, the next steps are pretty clear to me. I have a pretty good view of what I would like to see in a set of interface contracts (old term but still works) between the existing node-red UI nodes - a UIbuilder NR interface and the front end uibuilder interface. I know @TotallyInformation would probably prefer that we would just use the uibuilder-fs.js however at least in the flutter case this is not possible. If flutter is truly the exception I am ok with this just being documented in code but I still feel it is worth the intellectual effort to document it

If people are keen I would like to jump on a call to talk about scoping the document and then look at a divide and conquer approach @dceejay I would think that this is a useful step regardless of technology for the next steps with the dashboard.

Over to you guys. I will keep hacking the POT a bit to add the data-intensive widgets and tidy up a bit by NR interface -Quite possibly I might end up deploying the flutter stuff for some internal dashboards at work as multi-user should be relatively straight forward. (and I could justify some work hours on it)

BTW if there are any flutter people out there who would like to help (i was left without a FE design bone), would love to hear from you

[BartButenaers]

Hello @chrisn-au,
That looks very promising. Please keep up the good work!

@chrisn-au

we do not want to send code

I'm not really experienced with frontend frameworks, so I was wondering what you guys mean with not sending code to the client. Is that only related to sending live code via websockets, or more general? Some examples:

• Does this mean it is not possible anymore to write custom UI/Javascript/CSS in a Template node? Hopefully this is not a very stupid question ...
• My UI nodes contain a large amount of client side code, quite heavily depending on the AngularJs scope variables. Is there an easy way for me to adapt my ui nodes in the near future to run your dashboard, or do I need to start again from scratch?

About custom UI nodes, I have another question. The current dashboard has a lot of parameters:

ui.addWidget({
                    emitOnlyNewValues: false,
                    forwardInputMessages: false,
                    storeFrontEndInputAsState: false,
                    persistantFrontEndValue: false,

Will those still exist? I assume not. Just asking because I have spend a LOT of free time in the past to optimize those settings, and hope not having to go through that hell again if I ever have to migrate my UI nodes ....

@chrisn-au

android have not tested android but should just work also.

Is that a web or native app? If the latter, do you think we could make use of functionality like push notifications, getting geographic location ... like other native apps?

[chrisn-au]

@BartButenaers

• Does this mean it is not possible anymore to write custom UI/Javascript/CSS in a Template node? Hopefully this is not a very stupid question ...
• My UI nodes contain a large amount of client side code, quite heavily depending on the AngularJs scope variables. Is there an easy way for me to adapt my ui nodes in the near future to run your dashboard, or do I need to start again from scratch?

Bart,

great questions

I have zero experience with the

@BartButenaers

custom UI/Javascript/CSS in a Template node

The flutter approach would NOT support this. Not to say there are not alternatives but you will not be able to send JS/UI/CSS in a template node in the flutter path. This may be a reason why the approach dies. I would expect a more HTML approach may still be able to support this but I am also out of my depth

@BartButenaers

My UI nodes contain a large amount of client-side code, quite heavily depending on the AngularJs scope variables. Is there an easy way for me to adapt my ui nodes in the near future to run your dashboard, or do I need to start again from scratch?

Migration of existing Dashboards will need to be a consideration. I have no answers now though

@BartButenaers

ui.addWidget({
                    emitOnlyNewValues: false,
                    forwardInputMessages: false,
                    storeFrontEndInputAsState: false,
                    persistantFrontEndValue: false,

This is an interface we need to get documented regardless of approach, It needs to be consistent and logical (not saying the current approach isn't - but the documentation does need to be)

In the flutter app approach, it will be native IOS Android, MacOs Windows, Linux, Web. It will support notifications, geo stuff etc. I suspect better than some of the Web technologies today

From my point of view, if the flutter code-gen approach has a role, it will be for the user who just wants to use core widgets out of the box and accept most layout restrictions or the POWER user who wants to write their own front end. The segment in the middle may be underserved and hence the whole approach may not be feasible