Http request node: requestTimeout is connectTimeout in reality (NR v3.0.2)

Hi There!

I just discovered something about the http request node and I'm wondering whether that might change with NR 3.1.x

The requestTimeout value is in fact the connect timeout, i.e. the limit on connecting to the server, it is not the time limit for the server to respond. This means that a server can open the connection within the requestTimeout time but then not respond for X seconds. The http request node would hold the connection for those X seconds and longer.

This flow is a demonstration of that - requestTimeout is 1 second while the endpoint delays 5 seconds before sending reponse:

[{"id":"4de89aba64f6a744","type":"inject","z":"e46be6066ea2231c","name":"set url and count","props":[{"p":"payload"},{"p":"url","v":"http://localhost:1880/end/point","vt":"str"},{"p":"count","v":"30","vt":"num"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":428,"y":365,"wires":[["c5071dcbcf7ced38"]]},{"id":"c5071dcbcf7ced38","type":"function","z":"e46be6066ea2231c","name":"create count number of messages \\n with requestTimeout","func":"msg.payload = [];\n\nfor ( var idx = 0 ; idx < msg.count; idx++ ) {\n    msg.payload.push({\n        url: msg.url,\n        requestTimeout: 1000,\n    })\n}\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":690,"y":270,"wires":[["bfb961534391079d"]]},{"id":"bfb961534391079d","type":"split","z":"e46be6066ea2231c","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":948,"y":340,"wires":[["b17afc78fb472d40"]]},{"id":"b17afc78fb472d40","type":"http request","z":"e46be6066ea2231c","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[],"x":1116,"y":395,"wires":[["5f78cbafd3481358"]]},{"id":"e6be7a5009f301b6","type":"catch","z":"e46be6066ea2231c","name":"","scope":["b17afc78fb472d40"],"uncaught":false,"x":1143,"y":365,"wires":[[]]},{"id":"5f78cbafd3481358","type":"join","z":"e46be6066ea2231c","name":"","mode":"custom","build":"array","property":"","propertyType":"full","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":1301,"y":444,"wires":[["ebfe60187962ec1d"]]},{"id":"68b5f5cc96af4dfc","type":"http in","z":"e46be6066ea2231c","name":"","url":"/end/point","method":"get","upload":false,"swaggerDoc":"","x":564,"y":516,"wires":[["b3176bfe3e920a9a"]]},{"id":"d4f4d119f907475a","type":"http response","z":"e46be6066ea2231c","name":"","statusCode":"","headers":{},"x":980,"y":542,"wires":[]},{"id":"ebfe60187962ec1d","type":"function","z":"e46be6066ea2231c","name":"group by statusCode","func":"var newpload = {};\n\nfor ( var idx = 0; idx < msg.payload.length; idx++ ) {\n    (newpload[msg.payload[idx].statusCode] ||= []).push(msg.payload[idx])\n}\n\nmsg.payload = newpload;\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":1530,"y":444,"wires":[["cfd421c114b151c1"]]},{"id":"cfd421c114b151c1","type":"debug","z":"e46be6066ea2231c","name":"debug 43","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":1774,"y":389,"wires":[]},{"id":"b3176bfe3e920a9a","type":"delay","z":"e46be6066ea2231c","name":"","pauseType":"delay","timeout":"5","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1700","randomLast":"2000","randomUnits":"milliseconds","drop":false,"allowrate":false,"outputs":1,"x":784,"y":582,"wires":[["d4f4d119f907475a"]]}]

I had a look at the code and it seems that the timeout opt is set for the got request. In newer versions of the got module timeout is a hash allowing for fine-grained timeout specifications, i.e., connect v. response timeout which would prevent this behaviour from the client side.

I was wondering whether new version of NR would support that?

No one has proposed any changes in this area, so the behaviour will be the same as it is now.

Additionally, I suspect those options were added after they dropped support for CommonJS from v12 (now fully esm).

And while it may be possible to upgrade via dynamic import, for now, Node-RED is sticking with got V11.x as we use node require.

I have to admit that I only came across of this because I was trying to create a benchmarking flow similar to apache bench (ab), perhaps that makes this a bit of an edge case.

I created a work around using another delay (on the client side) - for benchmarking this solution is ok I guess, no idea whether it should be used for production.

Hack-around flow:

[{"id":"4de89aba64f6a744","type":"inject","z":"e46be6066ea2231c","name":"set url and count","props":[{"p":"payload"},{"p":"url","v":"http://localhost:1880/end/point","vt":"str"},{"p":"count","v":"300","vt":"num"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":370,"y":255,"wires":[["c5071dcbcf7ced38"]]},{"id":"c5071dcbcf7ced38","type":"function","z":"e46be6066ea2231c","name":"create count number of messages \\n with requestTimeout","func":"msg.payload = [];\n\nfor ( var idx = 0 ; idx < msg.count; idx++ ) {\n    msg.payload.push({\n        url: msg.url,\n        requestTimeout: 1000,\n    })\n}\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":421,"y":358,"wires":[["bfb961534391079d"]]},{"id":"bfb961534391079d","type":"split","z":"e46be6066ea2231c","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":637,"y":226,"wires":[["9c39e4c08eb3d981"]]},{"id":"b17afc78fb472d40","type":"http request","z":"e46be6066ea2231c","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[],"x":1194,"y":322,"wires":[["5f78cbafd3481358"]]},{"id":"e6be7a5009f301b6","type":"catch","z":"e46be6066ea2231c","name":"","scope":["b17afc78fb472d40"],"uncaught":false,"x":1221,"y":292,"wires":[[]]},{"id":"5f78cbafd3481358","type":"join","z":"e46be6066ea2231c","name":"","mode":"custom","build":"array","property":"","propertyType":"full","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":1518,"y":323,"wires":[["5c72a477a083347e"]]},{"id":"68b5f5cc96af4dfc","type":"http in","z":"e46be6066ea2231c","name":"","url":"/end/point","method":"get","upload":false,"swaggerDoc":"","x":500,"y":814,"wires":[["b3176bfe3e920a9a"]]},{"id":"d4f4d119f907475a","type":"http response","z":"e46be6066ea2231c","name":"","statusCode":"","headers":{},"x":977,"y":628,"wires":[]},{"id":"ebfe60187962ec1d","type":"function","z":"e46be6066ea2231c","name":"group by statusCode","func":"var newpload = {};\n\nfor ( var idx = 0; idx < msg.payload.length; idx++ ) {\n    (newpload[msg.payload[idx].statusCode] ||= []).push(msg.payload[idx])\n}\n\nmsg.payload = newpload;\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":1692,"y":585,"wires":[["1de2adbf5b60cbba"]]},{"id":"b3176bfe3e920a9a","type":"delay","z":"e46be6066ea2231c","name":"Random delay between 1.5 and 4 seconds","pauseType":"random","timeout":"5","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1500","randomLast":"4000","randomUnits":"milliseconds","drop":false,"allowrate":false,"outputs":1,"x":884,"y":814,"wires":[["f569c0f5674297ed"]]},{"id":"df6c6a244431f5a9","type":"delay","z":"e46be6066ea2231c","name":"","pauseType":"delay","timeout":"2500","timeoutUnits":"milliseconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"allowrate":false,"outputs":1,"x":1075,"y":188,"wires":[["175c314a82388d54"]]},{"id":"9c39e4c08eb3d981","type":"function","z":"e46be6066ea2231c","name":"ready, set, go","func":"\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":865,"y":323,"wires":[["b17afc78fb472d40","df6c6a244431f5a9"]]},{"id":"175c314a82388d54","type":"change","z":"e46be6066ea2231c","name":"","rules":[{"t":"set","p":"statusCode","pt":"msg","to":"TIMDOUT","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":1309,"y":188,"wires":[["5f78cbafd3481358"]]},{"id":"f569c0f5674297ed","type":"change","z":"e46be6066ea2231c","name":"","rules":[{"t":"set","p":"statusCode","pt":"msg","to":"$random() > 0.5 ? 404 : 200","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":883,"y":719,"wires":[["d4f4d119f907475a"]]},{"id":"5c72a477a083347e","type":"join","z":"e46be6066ea2231c","name":"wait for 2 messages","mode":"custom","build":"array","property":"","propertyType":"full","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"0.5","count":"2","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":1498,"y":442,"wires":[["c746c66762c1d0aa"]]},{"id":"c746c66762c1d0aa","type":"change","z":"e46be6066ea2231c","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload[0].payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":1532,"y":517,"wires":[["ebfe60187962ec1d"]]},{"id":"1de2adbf5b60cbba","type":"debug","z":"e46be6066ea2231c","name":"results of first resultset","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":1872,"y":492,"wires":[]}]

Sorry eating humble pie over here - I just realised that the flow I created had a subtle error. The http request node works as expected if I set the requestTimeout on the msg and not on the msg.payload - doh!

That was my mistake in the original flow example, if set correctly, the timeout will work as expected and kill the request if the server delays in responding.

I apologise for the confusion... mea culpa!

Not so - Node-RED 3.1 updates to the latest version of got using dynamic imports.

Oh, odd. I checked the Dev branch imports and seen it was still 11.x

Using my eyes properly I checked again and see it is 12.x

Still unsure if the timeouts object is a 12.x or 13.x thing - though it is moot at this point anyhow

Moving to a timeout object was one of the tasks of updating to got 12

1 Like

So I documented my thoughts on requestTimeout. In doing so I realised that the global setting httpRequestTimeout is probably sufficient for most use-cases and that explicitly setting requestTimeout might just be an edge-case.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.