JSON payloads larger than 100kB are refused when using UI-Builder

General
,
1

Hello,

We're currently using Node-RED 4.0.2 and we've upgraded UI-Builder from 3.3.1 to 6.8.2.

Since the upgrade, we are encountering 413 Payload Too Large errors when posting JSON payloads that are higher than 100kB:

PayloadTooLargeError: request entity too large
    at readStream (/usr/src/node-red/node_modules/raw-body/index.js:163:17)
    at getRawBody (/usr/src/node-red/node_modules/raw-body/index.js:116:12)
    at read (/usr/src/node-red/node_modules/node-red-contrib-uibuilder/node_modules/body-parser/lib/read.js:79:3)
    at jsonParser (/usr/src/node-red/node_modules/node-red-contrib-uibuilder/node_modules/body-parser/lib/types/json.js:138:5)
    at Layer.handle [as handle_request] (/usr/src/node-red/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/usr/src/node-red/node_modules/express/lib/router/index.js:328:13)
    at /usr/src/node-red/node_modules/express/lib/router/index.js:286:9
    at Function.process_params (/usr/src/node-red/node_modules/express/lib/router/index.js:346:12)
    at next (/usr/src/node-red/node_modules/express/lib/router/index.js:280:10)
    at expressInit (/usr/src/node-red/node_modules/express/lib/middleware/init.js:40:5)

This issue only happens after adding the UI-Builder dependency (without it, Node-RED will accept such large payloads).

Steps to reproduce:

  • Create a Node-RED settings.js file:
module.exports = {
    httpAdminRoot: '/node-editor',
    httpNodeRoot: '/node'
}
  • Create a DockerFile:
FROM nodered/node-red:4.0.2-20

USER node-red
RUN yarn add node-red-contrib-uibuilder@6.8.2

COPY ./settings.js /data/settings.js
  • Start Node-RED:
docker build .
docker run -p 1880:1880 <image-id>

Strangely enough, the error does not appear if the configured httpAdminRoot is equal to httpNodeRoot, in the Node-RED settings file.

The Node-RED flows don't even need to have any active UI-Builder node for the issue to happen.

Does anyone have ideas what could cause this issue?

Thanks

2

Welcome to the forum.

What/how are you posting? Not quite clear from your info.

3

Thanks!

Here are simple commands to reproduce the issue:

Working case:

  • Generate a working.json file (= 100kB)
echo "{\"k\":\"$(cat /dev/urandom | tr -dc A-Za-z0-9 | head -c 102392)\"}" > working.json
  • POST to Node-RED:
curl -H 'Content-Type: application/json' --data "@working.json" -X POST http://127.0.0.1:1880/node/test
  • This request reaches Node-RED (the 404 error is intended because no "http in" node listening to it)

Not working case:

  • Generate a notworking.json file (> 100kB)
echo "{\"k\":\"$(cat /dev/urandom | tr -dc A-Za-z0-9 | head -c 102393)\"}" > notworking.json
  • POST to Node-RED:
curl -H 'Content-Type: application/json' --data "@notworking.json" -X POST http://127.0.0.1:1880/node/test
  • Here I'm getting a 413 Payload Too Large response with the following response body:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>PayloadTooLargeError: request entity too large<br> &nbsp; &nbsp;at readStream (/usr/src/node-red/node_modules/raw-body/index.js:163:17)<br> &nbsp; &nbsp;at getRawBody (/usr/src/node-red/node_modules/raw-body/index.js:116:12)<br> &nbsp; &nbsp;at read (/usr/src/node-red/node_modules/node-red-contrib-uibuilder/node_modules/body-parser/lib/read.js:79:3)<br> &nbsp; &nbsp;at jsonParser (/usr/src/node-red/node_modules/node-red-contrib-uibuilder/node_modules/body-parser/lib/types/json.js:138:5)<br> &nbsp; &nbsp;at Layer.handle [as handle_request] (/usr/src/node-red/node_modules/express/lib/router/layer.js:95:5)<br> &nbsp; &nbsp;at trim_prefix (/usr/src/node-red/node_modules/express/lib/router/index.js:328:13)<br> &nbsp; &nbsp;at /usr/src/node-red/node_modules/express/lib/router/index.js:286:9<br> &nbsp; &nbsp;at Function.process_params (/usr/src/node-red/node_modules/express/lib/router/index.js:346:12)<br> &nbsp; &nbsp;at next (/usr/src/node-red/node_modules/express/lib/router/index.js:280:10)<br> &nbsp; &nbsp;at expressInit (/usr/src/node-red/node_modules/express/lib/middleware/init.js:40:5)</pre>
</body>
</html>
4

But surely that wouldn't show the error anyway? The data won't ever reach ExpressJS because there is not endpoint to handle it. You should add a http-in/-response endpoint and test with and without UIBUILDER installed. That will tell us whether it is the way that the Node-RED ExpresJS web service is able to handle it or not and is the first debugging step. Honestly, for this test, it should make no difference if UIBUILDER is installed or not.

If you look up the default limits for ExpressJS, you will see that it has a default maximum POST data size of 100kb.

If you want to submit more than that, you need to adjust the settings.

This is not related to UIBUILDER.

However, for UIBUILDER, you can set it to use a DIFFERENT ExpressJS server to Node-RED's default user-facing server if you want to. UIBUILDER's settings allow you then to configure Express however you like.

5

I have created the following Node-RED flow to listen on the endpoint I'm using (/test) and respond with a OK response body:
image

[{"id":"f6f2187d.f17ca8","type":"tab","label":"Flow 1","disabled":false,"info":""},{"id":"7b9a95dc7c2c44bf","type":"http in","z":"f6f2187d.f17ca8","name":"","url":"/test","method":"post","upload":false,"swaggerDoc":"","x":140,"y":60,"wires":[["1c1199b91adfbd15"]]},{"id":"0fa59f2a153b9007","type":"http response","z":"f6f2187d.f17ca8","name":"","statusCode":"200","headers":{},"x":520,"y":60,"wires":[]},{"id":"1c1199b91adfbd15","type":"change","z":"f6f2187d.f17ca8","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"OK","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":340,"y":60,"wires":[["0fa59f2a153b9007"]]}]

When using the commands from my last post, I'm getting a 200 OK with the working.json but still a 413 Payload Too Large with the notworking.json.

I agree that UI-Builder should not make any difference, but it turns out that by removing the UI-Builder dependency from my DockerFile, the error disappears for notworking.json.

Something tells me that the Express.js limit configuration that Node-RED uses (which I proved isn't the default 100kB) is actually erased by UI-Builder to the default value, which is unwanted.

6

OK, I will try to find some time to do some checking. Certainly, that should not happen but hey, Node-RED is complex! :slight_smile:

In the meantime, you might want to check out the uibuilder options in settings.js: UIBUILDER Documentation v7

If you configure a separate port in those, uibuilder will create its own instance of Express and shouldn't impact Node-RED.

Also, v7.1.0 is the current release of UIBUILDER. Each release does get bug-fixes as well as features so you should consider trying that version. Just check out the changes first: Release MAJOR RELEASE - now node.js 18+ · TotallyInformation/node-red-contrib-uibuilder · GitHub

7

Thanks for the quick answer!

I've tried version 7.1 and the bug appears there as well.

When looking at the UI-Builder code, we found this line which adds a global middleware (on /) that parses JSON, with the default configuration (the limit is set to 100kB).

this.app.use(express.json())

That could explain why UI-Builder is impacting Node-RED globally.

Modifying this line by adding a custom higher limit seems to fix the issue, but I'm not convinced it's the cleanest fix.

8

But that line is wrapped in if ( uib.customServer.isCustom === true ) { - so it is only run if using the custom server, not if using the default node-red server. Do you have a uibuilder section in settings.js? If so, can you share it?

This is what is run when using the default server:

Wait, my bad, lower down:

Looks like something has ended up in the wrong place.

Expect a fix and a new v7.1.1 release soon.

9

On my side I see that it's run every time, it's below that if/else.

Nope, I only have httpAdminRoot and httpNodeRoot configuration in this file.

10

That's great to hear!! :grinning_face: