Can I "host" stand-alone nodejs apps inside UIBuilder nodes? If so, should I?

[shrickus]

I think i'm trying to do something with UIBuilder that it is not meant to do... but i'll see if anybody else has tried something similar.

I have a nodejs app that has 2 express servers running on different ports -- 1 to server the ui pages, and 1 for all the /api/xxxxx backend ajax calls.

Normally, the project is "built" with webpack, and then both servers are started as stand-alone nodejs commands -- but i'd like to find a way of putting both of them inside a uibuilder node, called say myapp.

It has been easy enough to serve the /src and /dist UI files by just putting them under the uibroot directory -- but i'm having trouble finding a way to enable all the api.js logic (the express.use('/api/xxxxx') endpoints)

I realize that uibuilder is meant to enable the front-end UI to connect with the back-end calls written as node-red flows... so perhaps i'm swimming upstream here, trying to reuse the node-red express server and add my own endpoint urls behind the root url that uibuilder has enabled.

However, if i could do that, then the ui page /myapp/index.html could make its normal ajax calls to /myapp/api/xxxxx and run without many modifications... has anybody ever tried to do that? or know of a way to "add on" my api endpoints to uibuilder? Perhaps this is a chance for me to contribute a new feature back to the UIBuilder code... would anybody else find this useful?

[TotallyInformation]

@shrickus

1 to server the ui pages, and 1 for all the /api/xxxxx backend ajax calls.

Urm, you mean that you have reinvented Node-RED!

@shrickus

but i'm having trouble finding a way to enable all the api.js logic (the express.use('/api/xxxxx') endpoints)

Well the simple answer would be to use a standard Node-RED http-in/-out pair to define an /api/:id endpoint set.

@shrickus

trying to reuse the node-red express server and add my own endpoint urls behind the root url that uibuilder has enabled.

Yes, kind of struggling to see why this would be better/easier than using Node-RED itself?

@shrickus

then the ui page /myapp/index.html could make its normal ajax calls to /myapp/api/xxxxx and run without many modifications

uibuilder is generally geared up to do data over websockets (socket.io strictly speaking) rather than REST API's. Node-RED already does REST API's really well. You could easily set your http-in node to have a URL pattern of /myapp/api/:id, uibuilder wouldn't object or get in the way at all as long as you don't create a uibuilder web page with the matching URL. At least I think it wouldn't, I've not tested that scenario.

@shrickus

Perhaps this is a chance for me to contribute a new feature back to the UIBuilder code... would anybody else find this useful?

I think that we would firstly need to look at why this might be better than the http-in/-out nodes if I'm being honest. If there is a good reason, happy to consider it.

[shrickus]

Hey Julian - thanks for the thoughtful answers...

I hope you know by now that I'd love to be able to do as much of my work in Node-RED as possible -- but in this case there are dozens of APIs already written in a stand-alone express app, utilizing a new instance of express with routes set up like so...

const app = express();

app.get('/api/ping', (req, res) => {
    res.json({ success: true });
});

// Get Custom Settings
app.get('/api/custom-settings', (req, res) => {
    console.info('GET custom-settings');
    sessionService.getCustomSettings(req, res);
});
// Save Custom Settings
app.post('/api/custom-settings', (req, res) => {
    console.info('POST custom-settings');
    sessionService.saveCustomSettings(req, res);
});

... yada yada -- dozens of api call later ...

app.listen(API_PORT, () =>
    console.info(`API Server started @ http://${API_HOST}:${API_PORT}/api`)
).setTimeout(180000); // Max connection time of 3 minutes

Notice how the first step is to instanciate an express app, and the last step is to start it running...
The other 90% of the middle of the api.js existing code is what I'd like to be hosted relative to the existing route added based on the uibroot url. Preferably without rewriting it all as node-red flows. It seems analogous to the way you pass your Vue app into uibuilder, but I admit that I'm a bit confused by how all of that just "works".

I think what I'm asking for is a way to graft my pure nodejs api code into the uibuilder backend -- then, going forward, I can migrate existing apis into node-red flows as needed. Perhaps using a technique like you do in the middleware checking code:

        //#region ----- Set up ExpressJS Middleware ----- //
        /** Provide the ability to have a ExpressJS middleware hook.
         * This can be used for custom authentication/authorisation or anything else.
         */
        var httpMiddleware = function(req,res,next) { next() }
        /** Check for <uibRoot>/.config/uibMiddleware.js, use it if present. Copy template if not exists @since v2.0.0-dev4 */
        let uibMwPath = path.join(uib.configFolder, 'uibMiddleware.js')
        try {
            const uibMiddleware = require(uibMwPath)
            if ( typeof uibMiddleware === 'function' ) {
                httpMiddleware = uibMiddleware
            }    
        } catch (e) {
            log.trace(`[uibuilder:${uibInstance}] uibuilder Middleware failed to load. Reason: `, e.message)
        }

I tried putting some code to check if the req.path starts with "/api/", but since that runs with every request, it's not really scalable. Anyway, if this whole discussion goes against what uibuilder is trying to do, I can probably find another way. There just seems to be so much in place that I could use, with some clever packaging...

Steve

[TotallyInformation]

I actually think that the easy way to accomodate your API's would be to create a simple custom node.

Connecting to Node-RED's Express app is easy and you could probably just lift all of your code straight into the node's .js file.

@shrickus

the last step is to start it running

You wouldn't need that part of course since the listener is already running.

---

On the other hand, that wouldn't let you use the custom app option in uibuilder. So the idea of being able to add custom paths is interesting. It is absolutely possible to add middleware to a custom path so it should be possible to engineer.

@shrickus

Perhaps using a technique like you do in the middleware checking code:

It would need to be somewhat different because you need to pass in the path not just the function which is what the current middleware allows.

If you look at the vNext branch, things are a lot easier to parse. There is a function in libs/web.js called addMiddlewareFile that currently takes a reference to a node instance, require's an external file of a specific name and location then app.uses it to the uibuilder node's url path.

That function could be extended with optional parameters to allow alternate filenames and paths.

Currently, that function is called from the instanceSetup function which in turn is called from the main uibuilder.js nodeInstance function.

So for vNext anyway, it wouldn't be that hard to add. However, it would require you to pull apart your existing code so that the functions were separate.

Hmm, thinking it through, that would be hard to link everything up.

---

Another possible option would be to allow an Express router to be loaded from an external file. This could then be app.used as this.app.use( tilib.urlJoin(node.url, 'api/'), router).

That way, the router could be arbitarily complex and any paths you define would always be under <url>/api/.....

I think that would work.

Just be aware though, I won't be adding any new features to the v4 (main) codebase, the next release will be v5 (vNext) and still has some way to go before release. But the good news is that the web.js library is reasonably stable so I think that you could safely have a play with it.

Let me know what you think.

[shrickus]

@TotallyInformation

Another possible option would be to allow an Express router to be loaded from an external file. This could then be app.use d as this.app.use( tilib.urlJoin(node.url, 'api/'), router) .

That way, the router could be arbitarily complex and any paths you define would always be under <url>/api/.... .

I think that would work.

Nice! I just came to a similar conclusion tonight, after reading through the express.Router() documentation again (for probably the 3rd time). It looks like I can just add modules.export = router; line to the end of my app.js routes, and then if we could add a require('app.js'); in your uibuilder initialization code, all of the absolute api urls in my code (e.g. /api/ping) would be added relative to the uibroot url (i.e. /uiapp/api/ping).

I think this would be a great addition, allowing each uibuilder "app" to have it's own built-in api urls -- while still allowing us to run our apps in stand-alone mode under different ports (if that's still needed). I will see how far I can get with the sample code above, and let you know how it goes. Cheers!

[TotallyInformation]

OK, I suggest creating a new function in web.js that takes a node instance url and a url root path extension as parameters so that we can reuse it.

So if your uib instance has a url of test, and the url root path is api, the router use path will be <uib-url>/api/.

I also suggest that, for now, we restrict the name of the router file to load to be api.js. You will need to pick it up from the node insances custom folder which you will find in node.customFolder assuming that you call the function from instanceSetup and pass the node parameter through. You have access to fs-extra for file handling. Obviously the api.js file will not exist so that needs to be handled gracefully but you will see examples of that in the web.js file.

Later on, this might be extended with new additions to the Editor panel to allow multiple router files to be loaded to multiple paths. But for now, lets keep it simple because I've still lots of other changes to make before v5 can be published.

The vNext branch should work fine though so testing shouldn't be an issue. I try to keep my GitHub pushes such that the GitHub version is a (mostly) working version. I'll let you know if I have to make any significant changes to web.js.

[shrickus]

Success! Here is a summary of the changes I had to make to web.js

First, I've added an instance variable to hold the express router object:

        /** Reference to ExpressJS app instance being used by uibuilder
         * Used for all other interactions with Express
         */
        this.app = undefined
        /** Reference to ExpressJS server instance being used by uibuilder
         * Used to enable the Socket.IO client code to be served to the front-end
         */
        this.api = undefined
        /** Reference to optional ExpressJS app routes available to this instance
         * Used for back-end local api methods (to avoid CORS, and to aggregate external services)
         */
        this.server = undefined

Then, defined this new addInstanceApiRoutes method:

    /** (SHR) Add static ExpressJS routes for instance local api endpoints
     * @param {uibNode} node Reference to the uibuilder node instance
     */
    addInstanceApiRoutes(node) {
        // Reference static vars
        const log = this.log

        // Add any (optional) local api routes to this uibroot uri (e.g. /<uibroot>/api/status)
        let apiRoutes = path.join(node.customFolder, 'src', 'api.js') // for testing -- expose in config settings later
        log.info(`Checking for api routes: ${apiRoutes}`)
        try {
            // Check if local node folder contains api.js & if NR can read it - fall through to catch if not
            fs.accessSync(apiRoutes, fs.constants.R_OK)
            this.api = require(apiRoutes)

            // @ts-ignore
            let apiUrl = tilib.urlJoin(node.url) //, 'api')
            this.app.use(apiUrl, this.api)
            log.debug(`[uibuilder:web:addInstanceApiRoutes:${node.url}] Using local api.js routes: ${apiUrl} ...`)
            this.api.stack.forEach(api => {
                if (api.route) {
                    log.debug(`${api.route.path} ${Object.keys(api.route.methods)}`)
                }
            })
        } catch (e) {
            if (!e.toString().contains('NOENT')) {
                log.debug(e)
            }
            log.debug(`[uibuilder:web:addInstanceApiRoutes:${node.url}] No local api.js routes found in folder: ${node.customFolder}`)
        }
    } // --- End of addInstanceApiRoutes() --- //

Then, through much trial and error, found out that this needs to happen after the static instance route (otherwise they get swallow up):

        /** Serve up the uibuilder static common folder on `<url>/<commonFolderName>` (it is already available on `../uibuilder/<commonFolderName>/`, see _webSetup() */
        let commonStatic = serveStatic( uib.commonFolder, uib.staticOpts )
        // @ts-ignore
        this.app.use( tilib.urlJoin(node.url, uib.commonFolderName), commonStatic )

        // (SHR) Local API endpoints - Add routes to this node's root url, taken from api.js (if found)
        this.addInstanceApiRoutes(node)

Not sure how to expose some of this to the node's configuration UI... but at least I have something I can use to experiment. So far, it looks promising! Thanks again.

[TotallyInformation]

@shrickus

First, I've added an instance variable to hold the express router object:

Not sure that is correct? If it is an instance-level reference, then it should either be added to the node object or the class object needs to have a router for each instance (denoted by the instance URL which is the key).

@shrickus

path.join(node.customFolder, 'src', 'api.js')

Interesting, I thought it would go in path.join(node.customFolder, 'api.js')? Assumptions, assumptions!

It really isn't front-end code and so shouldn't be in the src or dist folders otherwise it will get served up to the front-end which could get nasty!

Maybe path.join(node.customFolder, 'libs', 'api.js') would be better?

---

Sorry, food delivery arrived - back later.

[shrickus]

All good points -- I'll adjust the code later tonight...

The stand-alone node.js app code uses src/server and src/client directories, so I just loaded the api.js file from the under src. Clearly not the right place to look for it, but to be really reusable that relative path would need to be configurable in the node itself, right?

[TotallyInformation]

@shrickus

to be really reusable that relative path would need to be configurable in the node itself, right?

I think that, for now, it is enough to have it in a fixed place (relative to the instance) with a fixed name. If people think it worth the effort, a future enhancement would be to add something extra to the Editor panel.

One of the things I'm working on is a rewrite of the package/library manager. With the new version, you will be able to install packages to a local instance folder, and the uibRoot folder. I'm going to try and gracefully stop using the userDir folder altogether.

Then I can use the package.json files in each location to keep the package metadata that uibuilder needs, the list of installed packages will, of course already be in the dependencies property of the package.json.

This has the added advantage that Templates will be able to define their own package dependencies (complete with versions). I will later add an auto-install to the Template handler. Easy to do now that I've broken the package management code into its own class as well.

So the first enhancement to your router loader is likely to be an extra property on the package.json that will let uibuilder load as many external route files as needed. And that will apply both to the common uibRoot level as well as at each node instance level.

[TotallyInformation]

Seems that I was, as usual, wildly optimistic about not needing to change the web.js class mobule

Instead, I've ended up doing some MAJOR work on it.

I've moved all of the messy route allocations to a much stronger structured set of Express Routers. This should really make route management a lot simpler and easier in the future. I was forever getting lost in all of the various routes.

The Admin and other API's have also been moved over with the v2 and v3 admin API's being in their own modules.

The v2 admin API's have been moved under the <httpAdminRoot>/uibuilder/ the same as the v3 API's which makes more sense and helps keep the Node-RED admin routes cleaner.

I've been able to tidy up the Express API use as well and that has resulted in 2 external packages going from the uibuilder dependencies.

I've added some new features to help with route debugging. I still need to update the details pages but they should make more sense once I've completed that.

I have been unpicking some of the package management that is still caught up in the web.js class - it should be in a new package management class but that code is so arcane, I've been struggling to follow it - even though I wrote it!

In v5, package management will be a VERY different beast. It will allow you to install pretty much anything you can install with npm including using @ scopes, GitHub packages and even local packages as well has been able to specify versions or branches.

Unfortunately, I think that will result in a breaking change in that installing front-end packages in the userDir folder was, in hindsight, not such a good idea. v5 will use the uibRoot folder which makes a lot more sense, especially now that you can move that folder to wherever you like. It also means that I'll be able to make good use of the package.json file and retire some of the other custom files.

As usual, I've managed to take myself down dozens of rabbit holes resulting in tearing apart a significant amount of the uibuilder code. But, despite the work, I'm feeling a lot happier with the result with a lot of badly evolved code being reworked.

I've pushed some updates to GitHub vNext branch and everything should still work though you may find the Node-RED log getting a bit busy.

I'll probably need to manually include your changes rather than trying to merge a PR but that's fine as it seems pretty straight-forward.

[shrickus]

Well I did pull in the latest vNext code last night, thanks for major push! You were right about how much cleaner (and more verbose!) it is now...

I was thinking about your original comments, on why not just using node-red flows to handle all the back-end api logic. Logically they are just 2 different languages for implementing the rest apis, so I feel there is still merit to enabling both "paths" for providing data to the front-end ui. But the "hidden" nature of the internal api routing feels very non-standard for node-red -- I'd love to be able to see something in the node-red editor that shows I'm using internal (coded) api logic... even if it's just a placeholder in my flows.

At the risk of sending you down another rabbit hole (!) what would you think about adding a third output port to the uibuilder node, which could be wired to 1 or more uibuilder-api nodes? This new api node could hold all the configuration that connects an existing nodejs express router into this uib instance (e.g. relative path like api/*, code-behind file like dist/server/api.js, etc). It seems that would keep the uibuilder node "pure" for the other 90% of the world that will never need this feature, while giving me a place to hang debug nodes and other instrumentation.

Anyway, I just wanted to propose an alternate solution, before the instance api work got too woven into the current node development. Of course, with all of the newly refactored code, it may be trivial to add to the existing node, which also works well for my use case. Let me know if you need me to provide any more code, or use cases -- I mean, I don't mind being a backseat driver throwing out directions, but I don't want you to feel like tossing me out on the side of the road! Cheers!

[TotallyInformation]

I'd be against having a 3rd output really. I think that the main uibuilder node is complex enough so I think it would be confusing for the 99.9% of people who wouldn't need to use it.

Also, if you are going to introduce a new node, what is the benefit again? Why not use the http-in/out nodes that will give you a very clear flow-based logic view?

As far as I can see, the only benefit to fitting your API to uibuilder is to be able to shove the logic in a node.js module rather than having it all showing in a flow.

So if you want to be able to see the logic n your flow, use http-in/out.

If you want some additional API's closely associated with your uibuilder-facilitated, data-driven UI, then being able to use an ExpressJS API middleware module stored with the rest of your uibuilder code makes some sense. But then it doesn't make much sense to then also want something showing in the Editor. What am I missing?

@shrickus

new api node could hold all the configuration that connects an existing nodejs express router into this uib instance

Seems like a lot of trouble to go to - creating a node just to hold some config? Really you only want an actual node if you are going to reuse it and have messages going either in, out or both. Would your proposed node do that? Even if it does, you could still very easily do that with http-in/out.

@shrickus

code-behind file like dist/server/api.js

Urm, no - the API definition file cannot go in the src or dist folders, they are just for front-end code. it would have to be in a different folder under the instance root folder.

With the big changes to package handling in vNext, one new feature being introduced is the use of the package.json file for localised settings. I've only done the one in uibRoot so far - which is where front-end libraries now live. But I'll be using the package.json files in the instance root folders as well. Not only for standard things such as running build scripts but for uibuilder-specific things such as listing the library dependencies for the instance (which allows templates to define their required environment and eventually will allow uibuilder to install the required packages for a template). So I envisage using the package.json file to define any required API middleware files for the instance as well - that too would allow templates to provide API's which I see as being a really powerful capability.

@shrickus

Of course, with all of the newly refactored code, it may be trivial to add to the existing node, which also works well for my use case.

Well, more trivial than it was, that's for certain. But lets get to the bottom of this suggestion before moving forward.

[TotallyInformation]

Hi @shrickus, I've started to try and implement this but I've hit a problem that I'd like your feedback on please.

Because the uibRoot folder no longer needs to be under the userDir folder, an api.js file won't necessarily find ExpressJS.

I had hoped that the content of the api.js file would be as simple as:

const express = require('express')

const myRouter = express.Router()

// Add what you like to myRouter

// Export the router
module.exports = myRouter

But if your installation of Node-RED and uibuilder is configured in certain ways, this will not work.

I can think of some ways around this but I don't know that any are ideal:

• Pass a reference to express into the module when requireing the module.
• Install express into the instance folder

There may be other and perhaps better ways but I can't think of them right now and it is time to go pick up the takeaway

Let me know what you think and whether there would be a preferred method.

[shrickus]

Hey Julian,

I was just reading about your latest changes -- for my use case, the express library will always be part of the server code, since it was a stand-alone app before adding it to uibuilder... but I think I see what you mean.

If someone had a standard uibuilder flow set up, and they wanted to add some api.js routes, they would need to require express and export a new router function. That requirement seems to me to imply that they would need to add the express library to their package.json in order to even build and/or test their apis... or are you wanting to allow them to simply write some routes in the api.js and just somehow expect them to run using only the installed uibuilder node_modules?

Pretty complex stuff -- but i've always felt that the better the node's design hides those complexities, the simpler it will be to use. It might be easier to discuss the pros/cons on either slack or zoom, if you are so inclined. Please feel free to send me a DM next time you are working on the vNext code.

--Steve

[TotallyInformation]

In case anyone else comes across this thread. The issue with needing to reference the ExpressJS module has been resolved. The final design works slightly differently but gives the same results without the need for the reference. So the instance API feature will be included with uibuilder v5. It will, however, be disabled by default as uncontrolled end-user access to internal api's is a security risk. Use with caution.