Express 5 for Node-RED: a branch with an Express 4 compatibility layer for nodes

Core Development
1

Hi all,

Picking up from the (now closed) thread "Is the new express library (version 5) usable with NodeRED?" and issue #5097.

In #5097 knolleary said the ideal would be "a compatibility mode available that would allow us to upgrade, but retain some of the removed v4 APIs with deprecation warnings", because of the large ecosystem of nodes built on Express 4. I've built that, on top of moving core to Express 5.3, and I'd like your view before raising a PR.

What the branch does (3 commits against current main):

  1. Moves core and the core nodes to Express 5.3. Wildcard and inline-regex routes are rewritten for the new path syntax, Express 4's "extended" query parser is kept as the default (so msg.req.query still expands a[b]=1), and HTTP In finds the router on either Express 4 or Express 5 hosts.
  2. Fixes the "stream is not readable" error reported in #5097 when Node-RED is embedded in an Express 5 app that parses bodies itself. I reproduced it (deploy returned 500); with the fix, deploys and HTTP In work.
  3. Adds an Express 4 compatibility layer on RED.httpNode and RED.httpAdmin. It translates unnamed wildcards (restoring req.params[0]), optional :param? routes, res.send(status), res.json(obj, status), res.sendfile(), req.param(), app._router and writes to req.query. Each one logs a one-time deprecation warning naming the route or API. It can be turned off with express4Compatibility: false. Users' existing HTTP In URLs like /files/* keep working through it too.

How much of the ecosystem this affects: I scanned the 204 most-downloaded community packages. 21 use something Express 5 breaks. 13 of those register a route Express 5 refuses, so the whole module fails to load. That includes node-red-dashboard, node-red-node-ui-table and node-red-contrib-sun-position, nearly all with the same pattern: RED.httpAdmin.get('/my-node/js/*', ...) plus res.sendFile(req.params[0]). The rest use res.send(status), app._router or write to req.query.

I installed six of them on the branch. With the compatibility layer off, five fail to load. With it on, all six load and serve their files, and the log tells the authors what to change. I'm happy to share the full list and how I scanned.

Tests: full suite passes apart from failures that need ssh-keygen, IPv6 or network access in my environment; 19 new tests for the new code.

Questions:

  1. Is this the kind of compatibility mode you had in mind?
  2. Which branch and release would you want it for: dev, aimed at the next major version?
  3. How long should the compatibility layer stay before it's removed or turned off by default?

Happy to open a draft PR so you can look at the code, and to adjust any of it.

2

Hi and welcome to your first post. :smiley:

It would help, I think, if you could list the contributed nodes that you've tested with. The 3 very critical ones, since they are complex and widely used, would the original node-red-dashboard, the newer Flowfuse dashboard and node-red-contrib-uibuilder.

3

Thanks, that's a good place to start. I've now tested all three, and updated the branch because uibuilder exposed a gap in the compatibility layer.

How I tested. I installed the nodes into a fresh user directory and ran the same flow on three setups:

  • stock Node-RED (main at 277bf6e2b, Express 4) as the baseline
  • the Express 5 branch with the compatibility layer on
  • the Express 5 branch with it off (express4Compatibility: false)

For each dashboard, a headless Chromium (Playwright) opened the page and checked two things: a value pushed from the flow every 2 s appears, and clicking a button sends a message back into the flow, which then shows on the page. For uibuilder, the browser waited for a message from the flow via uibuilder.get('msg'), then called uibuilder.send(). A function node stored the reply in global context, and I read it back through the admin API. I repeated everything after two full redeploys.

node-red-dashboard 3.6.6, @flowfuse/node-red-dashboard 1.33.0, node-red-contrib-uibuilder 7.7.4

Stock (Express 4) Branch, compat on Branch, compat off
node-red-dashboard: page, live value, button :white_check_mark: :white_check_mark: :white_check_mark: widgets work, but ui_base fails to register /ui_base/js/* (module shows an error)
FlowFuse Dashboard: page, live value, button :white_check_mark: :white_check_mark: :cross_mark: ui-base throws on /dashboard/*, so no dashboard
uibuilder: page, flow → browser, browser → flow :white_check_mark: :white_check_mark: :cross_mark: crashes on app._router.stack
All of the above after 2 full redeploys :white_check_mark: :white_check_mark: —

With compat on, the log shows one deprecation warning per pattern:

[express4-compat] Route '/dashboard/*' uses Express 4 path syntax. It has been registered as '/dashboard/{*__express4_wildcard_0}'. ...
[express4-compat] Route '/ui_base/js/*' uses Express 4 path syntax. ... (also /ui_base/css/*, /ui_base/gs/*)
[express4-compat] app._router was renamed to app.router in Express 5.

What uibuilder needed. My first run with compat on still failed for uibuilder. web.cjs walks app._router.stack and matches layers by layer.regexp.toString(), in removeRouter(), the vendor router and the routes summary. Express 5 layers have no regexp. The compatibility layer now restores layer.regexp on RED.httpNode/RED.httpAdmin layers, built exactly as Express 4's path-to-regexp 0.1 did. For example, /uibtest gives /^\/uibtest\/?(?=\/|$)/i. After that, uibuilder passes, including route removal on redeploy.

For uibuilder itself, longer term: matching layers by their handle (the router object uibuilder created for that instance) rather than by regexp would work on both Express 4 and 5 without the compatibility layer. Happy to send a PR for that if useful.

Other nodes tested (module loads and serves its editor resources; compat off → compat on):

Node Compat off Compat on
node-red-node-ui-table 0.4.5 fails to load loads, serves tabulator.js
node-red-contrib-sun-position 3.0.1 fails to load loads, serves htmlglobal.js
node-red-contrib-light-scheduler 0.0.19 fails to load loads, serves its JS
node-red-color 1.0.0 fails to load loads, serves its files (via the res.sendfile() shim)
node-red-contrib-web-worldmap 5.9.0 loads loads

I also embedded Node-RED in an Express 5 app that runs express.json() itself, the "stream is not readable" case from #5097. Deploy and HTTP In work on the branch.

Not covered yet: third-party widgets for either dashboard, uibuilder's custom-server mode (it brings its own Express 4, so it shouldn't be affected), and anything beyond basic widgets. Tell me what else you'd like me to test, and I can share the test flow and scripts.

The full suite passes apart from environment-only failures (ssh-keygen, IPv6, proxy). There are 21 new tests for the new code.

4

Thanks Ryan, this is excellent work.

I would be more than happy to see a PR. Much of the complexity in the web module stems from trying to keep a record of active routes (url's and mapped folders), something that Express has never done at all well with which is why I build a manual catalogue so that people can see what endpoints are available. (Something, I should note, that would be very handy for Node-RED core as well.)

Though if you submit it against the main (7.7.4) branch, I may need to manually transfer it to dev. I'll drop you a PM with more detail but I've been actively working on the API modules this last week with some fairly significant changes incoming (not all quite in GitHub yet).

Just note that it should still be using the same Express module so if Node-RED moved to v5, so would UIBUILDER. Still, if this works with Node-RED's Express instances, the custom server should also work.

Hopefully one of the core devs will join this thread soon as well.

5

Quick update for anyone following: the uibuilder change is now up as a PR against main, so it works natively on both Express 4 and 5, including custom-server mode: Support ExpressJS v5 as well as v4 in web.cjs by ryanmurdoc · Pull Request #662 · TotallyInformation/node-red-contrib-uibuilder · GitHub. Thanks Julian for the steer.

6

Hi @ryanmurdoc

Thanks for looking at this. A PR against the dev branch would be a good next step so we can look at what is involved here.

7

PR is up against dev: Upgrade to Express 5 with an Express 4 compatibility layer for nodes by ryanmurdoc · Pull Request #6024 · node-red/node-red · GitHub

It’s the same three commits: the Express 5 upgrade, the fix for “stream is not readable” when embedded, and the Express 4 compatibility layer (on by default, express4Compatibility: false to turn it off). The uibuilder side is in TotallyInformation/node-red-contrib-uibuilder#662. Happy to rework anything, including splitting the compatibility layer into its own PR if that’s easier to review.

8

Many thanks to Ryan for this work. I am just integrating it into some other changes I'm making and this will be part of the v8 release. v8 up's the baselines of Node-RED and node.js to Node-RED's v5 baseline. It will also include the usual collection of code improvements, bug fixes, documentation updates, etc. Hopefully not too far away from a release now.

9

Thank you for the great work! My package (fomcubes-react) is also supporting Express 5 now! :flexed_biceps:

10

glad it helped