Encoding used in the upgraded http-request node

Core Development
1

Hi folks,

In my early Node-RED days, I submitted a pull-request for the http-request node. Because NodeJs always converted the raw bytes it received to a string. If you are working for example with images, that is a real performance killer. Because at the end you want to have a Buffer in your flow. And yet a useless string conversion was executed by NodeJs:

raw bytes --> string --> byte buffer

By suppressing the bytes to string and back string to bytes buffer, things were becoming much faster.

I was intercepting my http requests last night, and I suddenly saw that the (low level) interceptor library returned my camera images as strings. Don't know if the interceptor library does the string conversion or not, but I started wondering if my contribution was still working fine.

However In the latest version (which uses the "got" library behind the scenes), my code has been removed. So might be because the "got" library does things entirely different, which make my code contribution obsolete. But I see the following in the "got" documentation:

image

And that parameter is not currently available in the updated http-request node, so I assume the default "utf8" encoding is being used unfortunately...

So I added this to the node:

image

But debugging the "got" code is very difficult, to check whether this has better performance. And my interceptor still returns a string, so I am now a bit confused whether this has improved anything or not :roll_eyes:

So I am a bit stuck. Does anybody has tips about how to continue with this?

Thanks !!
Bart

2

Hello Bart,

isn't this option already implemented ?
by default the reply is utf-8 unless you go and change it in the node's settings to

image

3

Hi @UnborN,
Yes indeed the "binary buffer" option is already there. If you are receiving e.g. snapshot images from an IP camera, you need to use that option.

But at the time being, I had this problem:

  1. The camera sends the snapshot image as bytes.
  2. So NodeJs puts that in a binary buffer.
  3. But then NodeJs converted the bufer to a string, which consumes a lot of cpu.
  4. Since I had the "binary buffer" option selected, the http-request node converted the string back to a buffer, which again consumed a lot of cpu.

So in my pull-request I simply told NodeJs not to convert the buffer to a string. By doing that, the output of my http request node was the original binary buffer. As a result I avoided two conversions. Which was really noticable in the cpu usage on an RPI3, due to the large amounts of data involved.

But my code was removed in the new http-request node, when it was updated to use the "got" library. So I assumed it is not required anymore, by using "got" or more recent NodeJs versions.

Yesterday I was redesigning my node-red-contrib-http-logger node to use another http interceptor library under the cover. And that library returned me the image as string in the response, which is not what I expected. So I started debugging, but it is unreadable Javascript code (generated from Typescript). So have not found whether NodeJs does the conversion to string or my new interceptor library.

So I started this discussion to collect some tips or ideas...

4

i see .. i understand the difference now .. makes sense.

[EDIT]
I looked a bit at the source of http-request and if i understand the code correctly, the reply from the request is indeed in Buffer and only converted to utf8 if the option is selected. I dont think the code does that round trip you described.

See lines 519

if i console.log(msg.payload) which is the res.body i get a buffer and later it gets converted based on the options.

5

Hi @BartButenaers

The encoding option for got only applies if its asked to return text - which is its default behaviour unless configured otherwise.

The docs you link to for the encoding parameter says:

To get a Buffer , you need to set responseType to buffer instead.

And as you can see here we do set responseType to buffer - so got does give us back a buffer.

And as @UnborN points to, we then do the final buffer->text conversion ourselves if needed.

6

Ok thanks for the feedback! I had overlooked that parameter. That indeed makes sense.

I now see here that the new http(s) interceptor library - that I started using this week- converts a Buffer in the response body to a string :woozy_face:.
That was why I was confused, and caused me started thinking that the http-request did something incorrectly...

So case closed. My compliments on the "request" to "got" migration!

7

For those interested in this topic (read: "nobody except me"): the same issue had been reported in the interceptors library already two days ago.

1 Like
8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.