New Table node for Dashboard v2.0

[omrid]

I have been developing a new custom node called ui-tabulator for dashboard v2.0, and about to release a beta version shortly. Following is a short summary of the node's functionality & configuration options. I welcome comments, ideas & suggestions before I publish it on GitHUb.

Note: the node comes in addition to, not replacing, dashboard v2.0's native vue-based table node.

ui-tabulator

The ui-tabulator custom node serves as a container for the popular Tabulator JavaScript package, for presentation & management of UI tables. The node follows the concept of the ui-table node of dashboard v1.0, but with significantly richer functionality.
At the moment, the node exposes a basic set of capabilities out of the vast feature-set of tabulator. As we move forward, the node can evolve to expose more tabulator capabilities, according to feedback from the user community.

General Overview

• The node serves as a smart wrapper containing a Tabulator (table) object. For the most part, it calls the Tabulator API as-is (as defined in the Tabulator Documentation).
• The node enables automatic instantiation of the table (with user-defined configuration & initial data), as well as dynamic table create/destroy in runtime.
• Interface to the node is through messages (regular Node-red msg objects). The msg specifies a command, and returns the table's response
• In addition, the node can send unsolicited event messages for selected table events (on-create, on-change etc.)

Note: Messages sent to the node are gauranteed to return unchanged (except for payload with returned data and an error property in case of failure). This allows calling the tabulator node from link-call nodes. However, if events are enabled, they need to be side-tracked and bypass the link-return node (as they do not have a caller return address)

• In addition to messaging, there is an option to call the table object API directly from other template nodes, in case the required functionality cannot be sent in a message (for example: setting a conditional-formatting expression, which cannot be serialized into a message without breaching security).
• By default, the table operates in shared mode, i.e. multiple concurrent clients see the same data image. In this mode, a snapshot of the current table data is retained in the Node-red datastore, and reloaded upon browser open, refresh etc.
  The node also supports a Multi-User mode, which allows per-client table-data & messaging.

Node Configuration

The node configuration properties (in the editor):

• Name, Group, Size: - same as in all dashboard nodes
• Initial Table Configuration: JSON object with all table & column definitions, and (optional) initial data
• Notifications: selection of table events to be sent
• Multi-user mode: (Y/N)
• CSS theme: selection of a tabulator theme (light, dark etc.)
• Table Id: optional unique Id for getting direct API access to the tabulator object from other dashboard template nodes
• Pass through message from input: will forward the incoming message as-is to the output port, in addition to the node'e response

Supported Commands

• General: createTable, destroyTable, saveToDatastore, clearDatastore
• Data update: setData, replaceData, updateData, addData, updateOrAddData, addRow, updateRow, deleteRow, updateOrAddRow, clearData
• Data Retrieval: getData, getDataCount, getRow, searchData, getSelectedData,
• Table Appearance: setStyle(cell/row/column/table), showColumn, setSort, setFilter, getFilters, addFilter, removeFilter, clearFilter
• Misc: selectRow, deselectRow, download

Note: It is possible to call additional, read-only tabulator APIs beyond the ones listed above. However, they will fail if their returned data is not serializable into the returned msg

Supported Events

• tableBuilt, tablePreDestroy, tableDestroyed
• rowClick, rowDblClick, rowTap, rowDblTap, rowTapHold
• rowAdded, rowUpdated, rowDeleted
• cellClick, cellDblClick, cellTap, cellDblTap, cellTapHold
• cellEdited
• dataChanged, dataFiltered

Node Dependencies

• Node-JS version >= 18
• Node-red version >= 3.10
• Node-red dashboard 2.0, version >= 1.6.0
• Tabulator version >= 6.2 (comes bundled in the node installation)

[thebaldgeek]

I was really confused and frustrated to see that FlowFuse had made a clear decision to abandon the table node in dash V2, so glad to see the community pick it up.
To me the table node is a core node of any dashboard.

With that said, after reading your post 4-5 times I don't understand much of it.
Looking forward to seeing the release and testing it.
Will be watching this thread for the announcement of when we can install it and test things out.

(Similar features to ui_table v1 but with responsive card view is my core need).

[joepavitt]

@thebaldgeek

I was really confused and frustrated to see that FlowFuse had made a clear decision to abandon the table node in dash V2

Just to offer some clarity here:

• Original Dashboard didn't have a ui-table node in its core collection.
• Dashboard 2.0 does have a ui-table node in it's core collection (docs) - albeit it isn't as rich and customisable as the third party table in original Dashboard
• The popular table widget for D1.0 that everyone used was a community node

[joepavitt]

I also detailed a ui-template with custom data table here which includes the justification on the decision we made on our core ui-table functionality

[JeromeVISUALS]

Excellent, can't wait !!

[JeromeVISUALS]

Hi omrid,

quick questions, about two features I use with the custom tabulator integration, and I'd like to know if that could work with your custom node in DB 2, or if I can manage to find another way to achieve that result :

• dataTree structure : https://tabulator.info/docs/5.5/tree#overview

• In another table, I feed a column with timestamp, and use a custom formatter using luxon. I use that directly in the column definition as it's the only way I found to still be able to sort table using that column ( sort is using timestamp value) :

Here is the code in template node :

            {
                title: "Time",
                field: "booking.scheduleInfo.startTimestamp",
                headerFilter: "input",
                formatter: function (cell, formatterParams) {
                    // Convert incoming milliseconds to Luxon DateTime and format
                    var milliseconds = cell.getValue();
                    var dateTime = luxon.DateTime.fromMillis(milliseconds);

                    // Format the datetime as desired (e.g., 'yyyy-MM-dd HH:mm:ss')
                    return dateTime.toFormat('dd-MM-yyyy HH:mm');
                }
                ,
                width: 100,
                sorter: function (a, b, aRow, bRow, column, dir, sorterParams) {
                    var dateA = a; // Assuming the timestamp is in milliseconds
                    var dateB = b; // Assuming the timestamp is in milliseconds
                    return dateA - dateB;

                }

Thanks in advance,

Jerome

[omrid]

Hi,
First, please note that I'm basing the node on the current (latest) Tabulator release, which is 6.2.

In theory, the full scope of Tabulator functionality should be available and supported by the node (including dataTree structure - I have'nt tested it but you probably will )

The main limitation we have is with Node-red messaging. We cannot serialize a function into a msg object (like you do for your formatter). Obviously, we could serialize the function into the message and then use eval(), but this would immediately disqualify the node (major security breach).

The classic 'Node-red approach' for this, is use a function node serving as a "reactor". For example, assume you want automatic conditional color formatting for a cell or row, according to some data value. You can register to receive data-change events and let the reactor return a respective tbSetStyle message for changing the color.

Another method I'm trying to implement, is to enable access to the table object from another template node (using Tabulator.findTable()), which will enable to call the Tabulator APIs directly (as you do today). I'm facing some issues with memory spaces etc., hope to be able to solve it.

Yet another approach, would be to pre-configure in the node a few popular manipulation functions (per community requests), and then let the node feed them to Tabulator upon messages specifying the API, function, args etc.

[TotallyInformation]

While I am no D2 expert, I can say that it IS possible to serialise a function, send it to the front end and then execute it. It just isn't easy.

You can serialise by using fnName,toString() manually in a function node.

To execute at the front-end would need some custom code in a ui_template. The source code for the uibuilder client library has examples.

[thebaldgeek]

For some reason I thought this table was based on vuetifyjs like the rest of the dashboard v2 parts, but it seems not.

Is this new node going to require anything to be installed on the device running Node-RED?

It seems that tabulator has a responsive card view table option that vuetify does not.
@omrid have you used the card view with tabulator at all?

[omrid]

@thebaldgeek

For some reason I thought this table was based on vuetifyjs like the rest of the dashboard v2 parts, but it seems not.

Is this new node going to require anything to be installed on the device running Node-RED?

The node (like any dashboard 2.0) is based on Vue. It is importing the Tabulator package into a Vue framework.

No additional installation is required - the relevant Tabulator files are included in the node.

@thebaldgeek

It seems that tabulator has a responsive card view table option that vuetify does not.
@omrid have you used the card view with tabulator at all?

I'm not aware of a reason why it shouldn't work, but have not tried it myself yet.

[omrid]

@TotallyInformation

You can serialise by using fnName,toString() manually in a function node.

Yes, initially I did exactly that (and actually it was working well). However, this immediately flags the node as unsafe (code injection is considered a big no-no) and disqualifies it for any professional production environment (which are regularly scanned for security breaches), hence I had to back off.

If there is a way to inject & deserialize functions without raising a security alert, please show me how.

[TotallyInformation]

All code injections come with a risk. That is true of at least Dashboard 1 because that does masses of code injection AFAIK. Not sure about D2. But any ui_template is injecting code and is "unsafe" in that sense. So from a safety perspective, you probably shouldn't be allowing Dashboard in production either.

But then loading an HTML file is also doing code injection by allowing <script> tags at all.

So really it isn't the fact that it injects code, it is whether or not you CONTROL that injection. No USER input should ever be accepted without sanitisation. Because UIBUILDER allows you to do all sorts of things, it also allows you to use the DOMPurify library (simply loading it is enough, the client automatically uses it if present).

Again, the danger here is allowing uncontrolled elements to be passed to the front end. That isn't just script but all manner of HTML and CSS as well as JS is "dangerous" if not controlled. I would, for example, never allow Node-RED in our production environments without some serious static checks and dynamic security along with strict controls over what it can and can't be used for.

You might want to try out UIBUILDER's front-end router with a route file that contains a script and see if that also triggers your security tooling since that uses a slightly different approach.

---

Oh, also, did you look at how the uibuilder client library applies script text? Check that it isn't different to the way you did it. For example, it does not use exec.

Here is how the uibuilder client applies script text:

    /** Attach a new text script to the end of HEAD synchronously
     * @param {string} textFn The text to be loaded as a script
     */
    loadScriptTxt(textFn) {
        const newScript = this.document.createElement('script')
        newScript.async = false
        newScript.textContent = textFn
        this.document.head.appendChild(newScript)
    }

As you can see, it adds a script tag to the DOM.

The FE router works differently because it loads external route fragments into a template tag (because there is an option to be able to remove the route from the DOM and later re-render without having to reload the resource). It then applies the template to the DOM - however, doing that does not activate any scripts in a template, so it does this:

    /** Remove/re-apply scripts in a container Element so that they are executed.
     * @param {HTMLElement} tempContainer HTML Element of container to process
     */
    _applyScripts(tempContainer) {
        const scripts = tempContainer.querySelectorAll('script')
        scripts.forEach( scr => {
            const newScript = document.createElement('script')
            newScript.textContent = scr.innerText
            tempContainer.append(newScript)
            scr.remove() // remove the origin
        })
    }

That is done before the template is rendered to the DOM. When the tempContainer is rendered, the scripts are run.

[omrid]

OK, thanks!
Maybe I'm missing something. Assuming I set the script into a DOM element as you show. How do I feed it (as a function) to the Tabulator API?
For example, assume I want to set a custom column formatter:

var tbl = Tabulator("#myTable",{
columns:  [
   {   title: "Name", field: "name", formatter: myCustomFormatter},
...

function myCustomFormatter(cell, formatterParams, onRendered)  {
    return "Mr. " + cell.getValue();
};

If I load myCustomFormatter as a script into the DOM element, will it be accessible as if I loaded it in a <script src=... tag?

[TotallyInformation]

@omrid

Assuming I set the script into a DOM element as you show. How do I feed it (as a function) to the Tabulator API?

I imagine you would use a ui_template node to send the function as text. To easily get the function as text, you can use a core template node with the output set to text. Or you could use a function node.

The ui_template node could also provide the conversion function - indeed, you could load that into the head - I assume the D2 can still do this?

I'm flying a bit blind there though as I generally don't use Dashboard 1 or 2 except for the occasional simple prototype.

@omrid

If I load myCustomFormatter as a script into the DOM element, will it be accessible as if I loaded it in a <script src=... tag?

Yes, it will. But remember, it will also be executed. So you will want to provide a function definition rather than raw code.

[omrid]

Thanks Julian @TotallyInformation,

I followed your advice, and it works perfectly! I even thought of a nice convention for passing these callback definitions in msg objects. We still face the challenge of protecting against malicious or poorly-written code (I can easily crash Node-red with bad code).

On second thoughts, maybe I should leave this out. Many (if not most) Node-red users prefer the "Low code" approach, and injecting callbacks will make things tricky and complex for them. Instead of callbacks, they can simply set a "reactor" node which acts upon Tabulator change notifications.

And those who are proficient in JS coding can very easily instantiate Tabulator in a template node, and work with it directly. I can prepare and share a base example.

@joepavitt, you are invited to share your thoughts on this.

[thebaldgeek]

I'd like to be as ready as I can be to test this new node as soon as possible.
@omrid could you please share how the msg.payload data has to be structured?
Either some debug screenshots, or code sample or output examples would be really helpful.
Thanks.

[omrid]

Here's an example from my testing environment:

And an example of how the node configuration looks like:

The node includes detailed documentation in the editor

@thebaldgeek

@omrid could you please share how the msg.payload data has to be structured?

When sending a msg to the node, msg.tbCmd indicates the required operation ('getData', 'updateData' etc.) and msg.tbArgs includes the required arguments.
In APIs where Tabulator is returning data, it is returned in msg.payload.
By default, payload returns the data received from tabulator as-is. For example, the 'getData' command, which calls the getData() Tabulator API, will return the following:

In some cases (e.g. getRow()) Tabulator API returns internal object structures, for which I do some additional parsing to extract the data to the returned msg:

@thebaldgeek

I'd like to be as ready as I can be to test this new node as soon as possible.

I am waiting for some answers (mainly around CSS issues) from the NR dashboard 2.0 developer (@joepavitt ) to conclude & publish the node. If you wish to help with testing, please give me your email address, and I'll send you the current node package & flow, so you can play with it

[Steve-Mcl]

@omrid

And an example of how the node configuration looks like:

How are you managing to run the old ACE editor? The Monaco editor has built in on-the-fly syntax checking and built-in formatting (i.e. no need for a custom function or button)

Is your source on a GitHub (or other public) repository?

[omrid]

@Steve-Mcl

How are you managing to run the old ACE editor? The Monaco editor has built in on-the-fly syntax checking and built-in formatting (i.e. no need for a custom function or button)

I took it from some example I found. Will be happy to use Monaco if it is better - could you point me to an example, or documentation?

[Steve-Mcl]

Are you using the built in APIs or are you importing ace editor yourself?

This would be far easier if you point us to your public repository? You do have your work safely published to a repository right?