JSON payloads larger than 100kB are refused when using UI-Builder

General
,
1

Hello,

We're currently using Node-RED 4.0.2 and we've upgraded UI-Builder from 3.3.1 to 6.8.2.

Since the upgrade, we are encountering 413 Payload Too Large errors when posting JSON payloads that are higher than 100kB:

PayloadTooLargeError: request entity too large
    at readStream (/usr/src/node-red/node_modules/raw-body/index.js:163:17)
    at getRawBody (/usr/src/node-red/node_modules/raw-body/index.js:116:12)
    at read (/usr/src/node-red/node_modules/node-red-contrib-uibuilder/node_modules/body-parser/lib/read.js:79:3)
    at jsonParser (/usr/src/node-red/node_modules/node-red-contrib-uibuilder/node_modules/body-parser/lib/types/json.js:138:5)
    at Layer.handle [as handle_request] (/usr/src/node-red/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/usr/src/node-red/node_modules/express/lib/router/index.js:328:13)
    at /usr/src/node-red/node_modules/express/lib/router/index.js:286:9
    at Function.process_params (/usr/src/node-red/node_modules/express/lib/router/index.js:346:12)
    at next (/usr/src/node-red/node_modules/express/lib/router/index.js:280:10)
    at expressInit (/usr/src/node-red/node_modules/express/lib/middleware/init.js:40:5)

This issue only happens after adding the UI-Builder dependency (without it, Node-RED will accept such large payloads).

Steps to reproduce:

  • Create a Node-RED settings.js file:
module.exports = {
    httpAdminRoot: '/node-editor',
    httpNodeRoot: '/node'
}
  • Create a DockerFile:
FROM nodered/node-red:4.0.2-20

USER node-red
RUN yarn add node-red-contrib-uibuilder@6.8.2

COPY ./settings.js /data/settings.js
  • Start Node-RED:
docker build .
docker run -p 1880:1880 <image-id>

Strangely enough, the error does not appear if the configured httpAdminRoot is equal to httpNodeRoot, in the Node-RED settings file.

The Node-RED flows don't even need to have any active UI-Builder node for the issue to happen.

Does anyone have ideas what could cause this issue?

Thanks

2

Welcome to the forum.

What/how are you posting? Not quite clear from your info.

3

Thanks!

Here are simple commands to reproduce the issue:

Working case:

  • Generate a working.json file (= 100kB)
echo "{\"k\":\"$(cat /dev/urandom | tr -dc A-Za-z0-9 | head -c 102392)\"}" > working.json
  • POST to Node-RED:
curl -H 'Content-Type: application/json' --data "@working.json" -X POST http://127.0.0.1:1880/node/test
  • This request reaches Node-RED (the 404 error is intended because no "http in" node listening to it)

Not working case:

  • Generate a notworking.json file (> 100kB)
echo "{\"k\":\"$(cat /dev/urandom | tr -dc A-Za-z0-9 | head -c 102393)\"}" > notworking.json
  • POST to Node-RED:
curl -H 'Content-Type: application/json' --data "@notworking.json" -X POST http://127.0.0.1:1880/node/test
  • Here I'm getting a 413 Payload Too Large response with the following response body:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>PayloadTooLargeError: request entity too large<br> &nbsp; &nbsp;at readStream (/usr/src/node-red/node_modules/raw-body/index.js:163:17)<br> &nbsp; &nbsp;at getRawBody (/usr/src/node-red/node_modules/raw-body/index.js:116:12)<br> &nbsp; &nbsp;at read (/usr/src/node-red/node_modules/node-red-contrib-uibuilder/node_modules/body-parser/lib/read.js:79:3)<br> &nbsp; &nbsp;at jsonParser (/usr/src/node-red/node_modules/node-red-contrib-uibuilder/node_modules/body-parser/lib/types/json.js:138:5)<br> &nbsp; &nbsp;at Layer.handle [as handle_request] (/usr/src/node-red/node_modules/express/lib/router/layer.js:95:5)<br> &nbsp; &nbsp;at trim_prefix (/usr/src/node-red/node_modules/express/lib/router/index.js:328:13)<br> &nbsp; &nbsp;at /usr/src/node-red/node_modules/express/lib/router/index.js:286:9<br> &nbsp; &nbsp;at Function.process_params (/usr/src/node-red/node_modules/express/lib/router/index.js:346:12)<br> &nbsp; &nbsp;at next (/usr/src/node-red/node_modules/express/lib/router/index.js:280:10)<br> &nbsp; &nbsp;at expressInit (/usr/src/node-red/node_modules/express/lib/middleware/init.js:40:5)</pre>
</body>
</html>
4

But surely that wouldn't show the error anyway? The data won't ever reach ExpressJS because there is not endpoint to handle it. You should add a http-in/-response endpoint and test with and without UIBUILDER installed. That will tell us whether it is the way that the Node-RED ExpresJS web service is able to handle it or not and is the first debugging step. Honestly, for this test, it should make no difference if UIBUILDER is installed or not.

If you look up the default limits for ExpressJS, you will see that it has a default maximum POST data size of 100kb.

If you want to submit more than that, you need to adjust the settings.

This is not related to UIBUILDER.

However, for UIBUILDER, you can set it to use a DIFFERENT ExpressJS server to Node-RED's default user-facing server if you want to. UIBUILDER's settings allow you then to configure Express however you like.

5

I have created the following Node-RED flow to listen on the endpoint I'm using (/test) and respond with a OK response body:
image

[{"id":"f6f2187d.f17ca8","type":"tab","label":"Flow 1","disabled":false,"info":""},{"id":"7b9a95dc7c2c44bf","type":"http in","z":"f6f2187d.f17ca8","name":"","url":"/test","method":"post","upload":false,"swaggerDoc":"","x":140,"y":60,"wires":[["1c1199b91adfbd15"]]},{"id":"0fa59f2a153b9007","type":"http response","z":"f6f2187d.f17ca8","name":"","statusCode":"200","headers":{},"x":520,"y":60,"wires":[]},{"id":"1c1199b91adfbd15","type":"change","z":"f6f2187d.f17ca8","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"OK","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":340,"y":60,"wires":[["0fa59f2a153b9007"]]}]

When using the commands from my last post, I'm getting a 200 OK with the working.json but still a 413 Payload Too Large with the notworking.json.

I agree that UI-Builder should not make any difference, but it turns out that by removing the UI-Builder dependency from my DockerFile, the error disappears for notworking.json.

Something tells me that the Express.js limit configuration that Node-RED uses (which I proved isn't the default 100kB) is actually erased by UI-Builder to the default value, which is unwanted.

6

OK, I will try to find some time to do some checking. Certainly, that should not happen but hey, Node-RED is complex! :slight_smile:

In the meantime, you might want to check out the uibuilder options in settings.js: UIBUILDER Documentation v7

If you configure a separate port in those, uibuilder will create its own instance of Express and shouldn't impact Node-RED.

Also, v7.1.0 is the current release of UIBUILDER. Each release does get bug-fixes as well as features so you should consider trying that version. Just check out the changes first: Release MAJOR RELEASE - now node.js 18+ · TotallyInformation/node-red-contrib-uibuilder · GitHub

7

Thanks for the quick answer!

I've tried version 7.1 and the bug appears there as well.

When looking at the UI-Builder code, we found this line which adds a global middleware (on /) that parses JSON, with the default configuration (the limit is set to 100kB).

this.app.use(express.json())

That could explain why UI-Builder is impacting Node-RED globally.

Modifying this line by adding a custom higher limit seems to fix the issue, but I'm not convinced it's the cleanest fix.

8

But that line is wrapped in if ( uib.customServer.isCustom === true ) { - so it is only run if using the custom server, not if using the default node-red server. Do you have a uibuilder section in settings.js? If so, can you share it?

This is what is run when using the default server:

Wait, my bad, lower down:

Looks like something has ended up in the wrong place.

Expect a fix and a new v7.1.1 release soon.

9

On my side I see that it's run every time, it's below that if/else.

Nope, I only have httpAdminRoot and httpNodeRoot configuration in this file.

10

That's great to hear!! :grinning_face:

11

Hello again,

While waiting for the 7.1.1 version, we'd like to make ourselves a temporary fixed version.

Do you think that moving the two lines you mentioned inside the if block would be a good quickfix?

Code for this part would look like this:

if ( uib.customServer.isCustom === true ) {

...

    this.app.use(express.json())
    this.app.use(express.urlencoded({ extended: true }))
} else {
    log.trace(`[uibuilder:web:_webSetup] Using Node-RED ExpressJS server at ${RED.settings.https ? 'https' : 'http'}://${RED.settings.uiHost}:${RED.settings.uiPort}${uib.nodeRoot === '' ? '/' : uib.nodeRoot}`)

    // Port not specified (default) so reuse Node-RED's ExpressJS server and app
    // @ts-expect-error
    this.app = /** @type {express.Application} */ (RED.httpNode) // || RED.httpAdmin
    this.server = RED.server
}

// Create Express Router to handle routes on `<httpNodeRoot>/uibuilder/`
this.uibRouter = express.Router(this.#routerOptions)

I'm really not familiar with Node.js development, so I prefer asking :stuck_out_tongue: Thanks!

12

Hi, thanks for the prompt. I have been hesitating about committing exactly that change because it isn't clear to me what the impact might be on existing users.

Unfortunately, Node-RED still uses the older body-parser extension for Express. Since Express v4, that has no longer been the recommended approach.

What I would suggest would be to try it. I've rather a lot on my plate with work and personal commitments just now so I would very much welcome someone else being able to test.

I think the change will be fine. I will need to add a settings.js option to allow people to adjust the limit themselves at some point. But given that you are the only person to have spotted this issue, I'd say that isn't too large a priority. :grin:

Let me know if that does the job please and I will try to find the time to commit a proper change.

13

Thanks, I'll make some tests.

I'm also trying to understand exactly what those two lines are meant for. I've found the commit that added them (more than 2 years ago, it's been a while :slight_smile:):

Apparently it seems related to logging, the commit does not mention that it should be useful only in case a custom server is used.
Knowing this, I'm worried to break that logging feature if I move those two lines elsewhere.

14

It shouldn't break anything because the standard Node-RED server already includes the older equivalent using the body-parser component. But you can now understand my slight nervousness.

The good news is that, few people are currently using uibuilder to push data from a client to Node-RED over HTTP(s), it normally all happens over Socket.IO which has a separate setting for size related issues (messages are normally limited to 1MB). Fewer still want to push such large data back to the server. I'm not even sure whether file uploads would require such a large setting.

Anyway, thanks for the reminder of that update. It was the start of an alternative route for pushing logging data from the front-end back to the server. In truth, the comment is correct in that the remainder of that requirement is still on the backlog and now quite a way's down the priority list.

So this setting should only impact UIBUILDER users creating user-facing API's with UIBUILDER and only when using the custom web server option.

So I think we are still pretty safe with the change.