As long the helpers don't create any functionality the bare core needing user needs to suppress, I cant see any problems. The thing is - the users are at wide range of knowledge base so all meaningful helpers are very welcome thus must be also respected by any user. The thing lives in world of openness. So strict rules can't always to be applied.
For all the reasons previously discussed, this is not possible. It would work for you but not for anyone else. It needs to work for everyone.
Correct indeed.
As previously said, there is a lot of refactoring that can be done in uibuilder to make it better, faster, etc. But it won't be losing any features - at least right now.
Once refactoring has progressed, it may be possible to do more and with the work I'm doing on the Socket.IO part right now, it may well be possible to have a separate node that only provides Socket.IO. But there is so much more to do in other areas, I can't see it being top priority for a while.
Exposing a core module that just manages the socket.io connection doesn't mean there can't be a separate module that wraps a ton of conveniences on top. It's just more modular. For example, there couple be vue-focused wrapper and a separate react-focused wrapper.
See the final para of that last message.
Sorry for hijacking this thread, but when I designed GitHub - cinhcet/node-red-contrib-component-dashboard: Flexible dashboard for Node-RED based on web-components I have thought about a lot of the things you are discussing here.
- Dashboard widgets are web-components
- Each dashboard widget has its own node
- There is a mechanism that enables to define the appearance etc. of the widget from Node-RED (I am not using it, but it is there)
- There is a mechanism to create new components from existing ones (sub-components).
- You can indeed create a "meta" widget that defines a grid where you name the widgets that should be placed there. Then everything could be configurable from Node-RED, no coding needed. That would be a lot of work, but possible. I am not interested in doing that, however, since for me it is faster to just write the html code.
- Widgets can be published as NPM modules (web-component and node)
- Buffering, replay etc. is handled for you like in the old dashboard
- No dependency on any web-framework
Just for your info....
1 Like
Haha! This is the hijack thread 
I did look at your repo, but at the time I didn't want to build any of this stuff... And also, it's bare JavaScript, which I'm not ready for. I need more hand-holding... 
But it does look nice!
It's actually only once I formulated the stuff I did in the last 5-6 posts that I decided to try and build a dashboard...
That crazy dashboard of mine keeps evolving 
a big refactoring brought the code size down and it's all becoming much nicer and cleaner. I spent quite some time improving the editing and adding a few widgets:
Editing is now live, no longer a modal. Any changes happen immediately and the rest of the dashboard remain live too. Widgets can now be resized and moved, although the UI for that needs improvement (I just don't want to dive into drag and drop right now):
The way the editing of a widget works is that each of the component's properties can be bound to either a literal string, a literal number, or a data value coming in from node-red. The output of switches, buttons, etc can similarly be bound to a data value that goes back to node-red.
So far it all continues to look very promising but every single piece needs work! Except for some trouble I'm still having with the reactivity in Vue it's actually becoming functional 
The widgets themselves are surprisingly simple. They're basically just a remapping of props with a little bit of logic around the vuetify (or other) base components. The sparkline, for example, is less than 60 lines of (quite sparse) code.
Another interesting development is that the entire thing is completely independent of node-red. It could be used for anything that can send it data over the websocket. Could even be served up and fed by an esp32...
4 Likes
This is pretty much what I was expecting to achieve with Vue and uibuilder where uibuilder is just acting as the conduit. Vue does most of the heavy lifting for the front-end design. Node-RED provides the data via the uibuilder channel.
2 Likes
Time for a demo 
If you have a moment, head over to FlexDash and play. To edit stuff you need to turn on editing mode in the gear menu at the top-right. This demo runs stand-alone with random data generated internally, there is no server connection (and edits are not saved across browser refreshes). I did start a readme with some high level architecture info and I've tried to be generous with comments in the source. Feedback appreciated.
5 Likes
Looking seriously good
1 Like
I know you like uPlot but have you considered using pivottable library at GitHub - nicolaskruchten/pivottable: Open-source Javascript Pivot Table (aka Pivot Grid, Pivot Chart, Cross-Tab) implementation with drag'n'drop. ?
In my opinion it is more suitable for monitoring sensor data from network of devices with very little coding.It include tables and graphs in same package. And unlike other graphing software it is easy to render table or graph with drag and drop selectors. It could be standard widget for FlexDash so user do not need any other separate widget for graphor table. I use following code for my purpose:
<script type="text/javascript">
// This example loads sensor data from a ESP32 gateway which collect data from remote sensor devices and store it in a CSV file.
$(function(){
var renderers = $.extend(
$.pivotUtilities.renderers,
$.pivotUtilities.plotly_renderers,
$.pivotUtilities.d3_renderers,
$.pivotUtilities.export_renderers
);
Papa.parse("http://192.168.0.3/sensors.csv", {
download: true,
skipEmptyLines: true,
complete: function(parsed){
for (let i = 0; i < parsed.data.length; i ++) {
var d = new Date(parsed.data[i][0]*1000);
parsed.data[i][0] = (("00" + d.getFullYear()).slice(-2) + ("00" + (d.getMonth() + 1)).slice(-2) + ("00" +
d.getDate()).slice(-2) + ("00" + d.getHours()).slice(-2) + ("00" + d.getSeconds()).slice(-2));
parsed.data[0][0] = "Timestamp";
if (parsed.data[i][1] == 6) {
parsed.data[i][1] = "Livingroom";
} else if (parsed.data[i][1] == 16) {
parsed.data[i][1] = "Kitchen";
} else if (parsed.data[i][1] == 26) {
parsed.data[i][1] = "Bedroom1";
} else if (parsed.data[i][1] == 36) {
parsed.data[i][1] = "Bedroom2";
} else if (parsed.data[i][1] == 46) {
parsed.data[i][1] = "Bedroom3";
} else if (parsed.data[i][1] == 56) {
parsed.data[i][1] = "Bedroom4";
} else if (parsed.data[i][1] == 66) {
parsed.data[i][1] = "Bathroom1";
} else if (parsed.data[i][1] == 76) {
parsed.data[i][1] = "Bathroom2";
} else if (parsed.data[i][1] == 86) {
parsed.data[i][1] = "Bathroom3";
} else if (parsed.data[i][1] == 96) {
parsed.data[i][1] = "Bathroom4";
} else if (parsed.data[i][1] == 106) {
parsed.data[i][1] = "Laundry";
} else if (parsed.data[i][1] == 116) {
parsed.data[i][1] = "Boiler Room";
} else if (parsed.data[i][1] == 126) {
parsed.data[i][1] = "Workshop";
} else if (parsed.data[i][1] == 136) {
parsed.data[i][1] = "Garage";
} else if (parsed.data[i][1] == 146) {
parsed.data[i][1] = "Office";
} else if (parsed.data[i][1] == 156) {
parsed.data[i][1] = "Tank";
} else if (parsed.data[i][1] == 166) {
parsed.data[i][1] = "Solar";
} else {
parsed.data[i][1] = "Unknown";
parsed.data[0][1] = "Location";
}
//console.log(parsed.data[i][1]); // (13)arrays of values by sensor type
}
$("#output").pivotUI(parsed.data, {
rows: ["Location"], cols: ["Timestamp"],
aggregatorName: "List Unique Values",
vals: ["Temperature"],
rendererName: "Line Chart",
});
}
});
var dragging = function(evt) {
evt.stopPropagation();
evt.preventDefault();
evt.originalEvent.dataTransfer.dropEffect = 'copy';
$("body").removeClass("whiteborder").addClass("greyborder");
};
var endDrag = function(evt) {
evt.stopPropagation();
evt.preventDefault();
evt.originalEvent.dataTransfer.dropEffect = 'copy';
$("body").removeClass("greyborder").addClass("whiteborder");
};
var dropped = function(evt) {
evt.stopPropagation();
evt.preventDefault();
$("body").removeClass("greyborder").addClass("whiteborder");
parseAndPivot(evt.originalEvent.dataTransfer.files[0]);
};
$("html")
.on("dragover", dragging)
.on("dragend", endDrag)
.on("dragexit", endDrag)
.on("dragleave", endDrag)
.on("drop", dropped);
});
</script>
Content of sensors.csv file is as follows:
Timestamp,Location,Voltage,RSSI,Temperature,Humidity,Pressure,Light,Open/Close,Level,Presence,Motion,Custom
1621906615,26,6,-37,51,78,232,86,0,0,0,0,0
1621906647,6,6,-37,49,44,225,55,0,0,0,0,0
1621906678,6,146,-37,46,45,228,72,0,0,0,0,0
1621906691,26,6,-36,46,45,228,72,0,0,0,0,0
1621906723,6,146,-39,54,88,250,66,0,0,0,0,0
Dependencies are as follows:
Thanks.
Dependencies for above code:
</script>
<!-- external libs from cdnjs -->
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/jquery/1.11.2/jquery.min.js"></script>
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/jqueryui/1.11.4/jquery-ui.min.js"></script>
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min.js"></script>
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/jqueryui-touch-punch/0.2.3/jquery.ui.touch-punch.min.js"></script>
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/PapaParse/4.1.2/papaparse.min.js"></script>
<script src="https://cdn.plot.ly/plotly-basic-latest.min.js"></script>
<!-- PivotTable.js libs from ../dist -->
<link rel="stylesheet" type="text/css" href="[../dist/pivot.css](https://pivottable.js.org/dist/pivot.css)">
<script type="text/javascript" src="[../dist/pivot.js](https://pivottable.js.org/dist/pivot.js)"></script>
<script type="text/javascript" src="[../dist/d3_renderers.js](https://pivottable.js.org/dist/d3_renderers.js)"></script>
<script type="text/javascript" src="[../dist/plotly_renderers.js](https://pivottable.js.org/dist/plotly_renderers.js)"></script>
<script type="text/javascript" src="[../dist/export_renderers.js](https://pivottable.js.org/dist/export_renderers.js)"></script>
Hmm, to be honest, the list of dependencies says it all... You're dragging the whole plotting world in there! 
In comparison, uPLot is just 35KB without any dependencies (that means it does not have dependencies, not that they're not included in those 35KB).
But I really look forward to you producing a pivottable widget for FlexDash!! I'm not quite there yet, but I believe that both developing and publishing a widget for others can be made very, very easy. And using a published widget can be zero-install easy. The bottom line is: please use the widget with the plotting library you like most! 
I'm kind'a committed now to making FlexDash usable by any node-red user 
, but I will draw the proverbial line in the sand when in comes to widgets: I'll implement the ones I need and either FlexDash is compelling enough so others contribute theirs or, oh well, it wasn't compelling enough...
3 Likes
As an FYI, I started a new thread for FlexDash a while ago: FlexDash: towards a new dashboard for Node-Red
There's now a demo that connects to Node-Red using websockets that can be tried out without downloading or installing anything (tl;dr; FlexDash and follow the instructions)
1 Like
Ha, so I hit a bit of a snag on the subscriptions thing...
The way I implemented the topic tree is as a JSON data structure, so for example, you might have two sensors:
{ "sensors": [
{ "temperature": 60, "rh": 45 },
{ "temperature": 66, "rh": 55 }
] }
and then topics just walk down the JSON tree, for example, a widget could subscribe to /sensors/0/temperature and receive 60. The sensor itself might update its readings by sending a message like
{ topic: "/sensors/0", payload: {temperature: 61, rh: 44} }
but now I'm looking at MQTT... that model doesn't carry over at all... The MQTT topic tree is not a data structure where one can update a whole subtree.
I just checked and mosquitto not only supports websockets but can also serve up static files over http, so it's totally feasible to serve up FlexDash from mosquitto and use it straight with MQTT without NR being involved at all. It's also feasible for FlexDash to connect both to mosquitto and to node-red and pull raw data straight from mosquitto and pull "processed data" from node-red. Given that all my sensors produce MQTT which then goes to NR this sounds like a pretty nice shortcut for some stuff.
Yikes, I just searched for "postgres websockets" and there's stuff there too! Chart widget pulling straight from the DB... Hmmmm.
But first I suppose I need to rework the pub/sub stuff to use "proper" MQTT topics?
1 Like
No, you have to have a function that does that. However, MQTT is so efficient that doing it even at scale isn't an issue. I've now taken to updating most of my home automation data structures in a way that deconstructs them to MQTT as well so that I can use event-driven flows.
It is also common to have both the deconstructed topics (where each topic leaf has a single value) AND a master topic containing the same data as a single JSON structure (for a good example of this, see Zigbee2MQTT).
This is along the same vein as de-normalising SQL tables.
I have to say that I think this is not a good idea. You are forcing the broker into a role that it isn't designed for and that not all brokers will be able to deliver (nor deliver in the same way). You are also creating greater architectural complexity where it isn't needed.
For serving up HTML and related static resources, use a web server. For serving dynamic resources, use a data-driven server such as Node.js (with ExpressJS or similar), PHP, Python (with a suitable library), PERL (
) or a higher-level tool like Node-RED.
Leave your broker as a Message Queue server which is what it is designed for.
Same problem, don't add unnecessary architectural complexity by making a database management engine do web serving!
Have servers do what they are good at and choose an appropriate interconnection protocol that is as efficient, standard and widely used as possible.
Node-RED is good at gluing things together. Mosquitto is good at handling a lightweight message queue, NGINX/Caddy/etc are good at serving HTTP and related resources. ...
Indeed there are good reasons why databases are designed one way, brokers another, and web servers yet another... - but hey - nothing ever progressed without pushing a few boundaries... I do recall having a great discussion back in the day with some gurus who shall rename nameless about whether it would be possible to create an "engine" that had three interfaces to it - namely, web, database and broker - such that you could send things to it and receive things from it and it would be transparent to each. IE it would handle subtree updates as per above - be queryable at different levels, etc.... Was a good discussion. Of course the database guy wanted to start from a database and add the rest, likewise the broker guy said we should start fro a broker... ho hum..
1 Like
Always start with the data, everything else is a bonus 
At this point I'm not sure whether pulling data from MQTT ends up being really nice or a can of worms. In the troubleshooting use-cases of a dashboard it's nice to be able to get raw data from the sensors before it gets processed. But on the other hand, it's raw and can be misleading or incomplete. Seems worth exploring...
In the whole "how do you feed data to the dashboard" question the biggest issue in my mind is still historical data. The issue ranges from having a couple of past data points for a sparkline, to showing the past data for a chart when the dashboard is loaded, to being able to retrieve historical data based on zooming and panning in a chart. There are a bunch of ways to implement this stuff, but none really feel right to me yet. The biggest issue being too much coupling between dashboard and server, e.g., the server side second-guessing how much data to save and ship on-connect when that really depends on widget config or screen width, and on whether the relevant widget is even visible.
Another issue that has been bugging me forever with the std dashboard is how sliders and switches should behave. Specifically, seeing how responsive FlexDash is, I'm thinking the best is for these things to just send a message and perhaps transition visually for 1-2 seconds and then snap back if a value change "confirmation" hasn't been received from the server. I.e. decouple the indicator part from the actuator part. This way if the server doesn't respond promptly then the user immediately sees that the action "didn't stick".
I know the std dashboard provides a couple of different options, but my experience with that has been rather negative. I remember clearly that when first confronted with the options I was confused even though I clearly understood the mechanics. I just could not figure out which way was going to work best. IMHO, it would be nice to pick one way and eliminate the choice.
