# Encoding used in the upgraded http-request node

**URL:** <https://discourse.nodered.org/t/encoding-used-in-the-upgraded-http-request-node/52380>\
**Category:** Core Development\
**Created:** [18 October 2021 22:14 UTC](https://discourse.nodered.org/t/encoding-used-in-the-upgraded-http-request-node/52380 "2021-10-18T22:14:12Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [18 October 2021 22:14 UTC](https://discourse.nodered.org/t/encoding-used-in-the-upgraded-http-request-node/52380/1 "2021-10-18T22:14:12Z")

</div>

Hi folks,

In my early Node-RED days, I submitted a [pull-request](https://github.com/node-red/node-red/pull/1176) for the http-request node. Because NodeJs always converted the raw bytes it received to a string. If you are working for example with images, that is a real performance killer. Because at the end you want to have a Buffer in your flow. And yet a useless string conversion was executed by NodeJs:

```auto
raw bytes --> string --> byte buffer

```

By suppressing the bytes to string and back string to bytes buffer, things were becoming much faster.

I was intercepting my http requests last night, and I suddenly saw that the (low level) interceptor library returned my camera images as strings. Don't know if the interceptor library does the string conversion or not, but I started wondering if my contribution was still working fine.

However In the latest version (which uses the "got" library behind the scenes), my code has been removed. So might be because the "got" library does things entirely different, which make my code contribution obsolete. But I see the following in the "got" [documentation](https://github.com/sindresorhus/got/blob/a485234df6b741393310f7341c256fa43aa5d3a4/documentation/2-options.md#encoding):

![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/f/c/fc1c7cc0ecaa6935ffcc8f695994f2ab98bc20eb.png)

And that parameter is not currently available in the updated http-request node, so I assume the default "utf8" encoding is being used unfortunately...

So I added this to the node:

![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/c/0/c04a5c97ea053d209eb46ba444833c633f592b49.png)

But debugging the "got" code is very difficult, to check whether this has better performance. And my interceptor still returns a string, so I am now a bit confused whether this has improved anything or not 🙄

So I am a bit stuck. Does anybody has tips about how to continue with this?

Thanks !!  
Bart

---

<div class="post-metadata">

**Author:** ![UnborN](https://avatars.discourse-cdn.com/v4/letter/u/4491bb/32.png) [@UnborN](https://discourse.nodered.org/u/UnborN)\
**Post date:** [19 October 2021 01:44 UTC](https://discourse.nodered.org/t/encoding-used-in-the-upgraded-http-request-node/52380/2 "2021-10-19T01:44:44Z")

</div>

Hello Bart,

isn't this option already implemented ?  
by default the reply is utf-8 unless you go and change it in the node's settings to

![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/a/8/a85400264368253d2fb635765a91153cc4e56efa.png)

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [19 October 2021 05:28 UTC](https://discourse.nodered.org/t/encoding-used-in-the-upgraded-http-request-node/52380/3 "2021-10-19T05:28:32Z")

</div>

Hi @UnborN,  
Yes indeed the "binary buffer" option is already there. If you are receiving e.g. snapshot images from an IP camera, you need to use that option.

But at the time being, I had this problem:

1. The camera sends the snapshot image as bytes.
2. So NodeJs puts that in a binary buffer.
3. But then NodeJs converted the bufer to a string, which consumes a lot of cpu.
4. Since I had the "binary buffer" option selected, the http-request node converted the string back to a buffer, which again consumed a lot of cpu.

So in my pull-request I simply told NodeJs not to convert the buffer to a string. By doing that, the output of my http request node was the original binary buffer. As a result I avoided two conversions. Which was really noticable in the cpu usage on an RPI3, due to the large amounts of data involved.

But my code was removed in the new http-request node, when it was updated to use the "got" library. So I assumed it is not required anymore, by using "got" or more recent NodeJs versions.

Yesterday I was redesigning my node-red-contrib-http-logger node to use another http interceptor library under the cover. And that library returned me the image as string in the response, which is not what I expected. So I started debugging, but it is unreadable Javascript code (generated from Typescript). So have not found whether NodeJs does the conversion to string or my new interceptor library.

So I started this discussion to collect some tips or ideas...

---

<div class="post-metadata">

**Author:** ![UnborN](https://avatars.discourse-cdn.com/v4/letter/u/4491bb/32.png) [@UnborN](https://discourse.nodered.org/u/UnborN)\
**Post date:** [19 October 2021 05:49 UTC](https://discourse.nodered.org/t/encoding-used-in-the-upgraded-http-request-node/52380/4 "2021-10-19T05:49:41Z")

</div>

> [@BartButenaers](#):
>
> So in my pull-request I simply told NodeJs not to convert the buffer to a string. By doing that, the output of my http request node was the original binary buffer.

i see .. i understand the difference now .. makes sense.

**[EDIT]**  
I looked a bit at the source of **http-request** and if i understand the code correctly, the reply from the request is indeed in Buffer and only converted to utf8 if the option is selected. I dont think the code does that round trip you described.

[See lines 519](https://github.com/node-red/node-red/blob/master/packages/node_modules/%40node-red/nodes/core/network/21-httprequest.js#L519)

if i `console.log(msg.payload)` which is the `res.body` i get a buffer and later it gets converted based on the options.

---

<div class="post-metadata">

**Author:** ![knolleary](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/knolleary/32/3_2.png) [@knolleary](https://discourse.nodered.org/u/knolleary)\
**Post date:** [19 October 2021 09:30 UTC](https://discourse.nodered.org/t/encoding-used-in-the-upgraded-http-request-node/52380/5 "2021-10-19T09:30:42Z")

</div>

Hi @BartButenaers

The `encoding` option for `got` only applies if its asked to return text - which is its default behaviour unless configured otherwise.

The docs you link to for the `encoding` parameter says:

> To get a [`Buffer`](https://nodejs.org/api/buffer.html), you need to set [`responseType`](https://www.npmjs.com/package/got#responseType) to `buffer` instead.

And as you can see [here](https://github.com/node-red/node-red/blob/master/packages/node_modules/@node-red/nodes/core/network/21-httprequest.js#L182) we do set `responseType` to `buffer` - so `got` does give us back a buffer.

And as @UnborN points to, we then do the final buffer-\>text conversion ourselves if needed.

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [19 October 2021 15:29 UTC](https://discourse.nodered.org/t/encoding-used-in-the-upgraded-http-request-node/52380/6 "2021-10-19T15:29:57Z")

</div>

> [@knolleary](#):
>
> And as you can see [here](https://github.com/node-red/node-red/blob/master/packages/node_modules/@node-red/nodes/core/network/21-httprequest.js#L182) we do set `responseType` to `buffer` - so `got` does give us back a buffer.

Ok thanks for the feedback! I had overlooked that parameter. That indeed makes sense.

I now see [here](https://github.com/mswjs/interceptors/blob/16fd35622efe6328497ade15e21a84f6afda0a93/src/interceptors/ClientRequest/createClientRequestOverride.ts#L146) that the new http(s) interceptor library - that I started using this week- converts a Buffer in the response body to a string 🥴.  
That was why I was confused, and caused me started thinking that the http-request did something incorrectly...

So case closed. My compliments on the "request" to "got" migration!

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [19 October 2021 15:40 UTC](https://discourse.nodered.org/t/encoding-used-in-the-upgraded-http-request-node/52380/7 "2021-10-19T15:40:54Z")

</div>

For those interested in this topic (read: "nobody except me"): the same [issue](https://github.com/mswjs/interceptors/issues/158) had been reported in the interceptors library already two days ago.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [2 November 2021 15:41 UTC](https://discourse.nodered.org/t/encoding-used-in-the-upgraded-http-request-node/52380/8 "2021-11-02T15:41:24Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.