# Http request node - gzip encrypted response

**URL:** <https://discourse.nodered.org/t/http-request-node-gzip-encrypted-response/30300>\
**Category:** General\
**Created:** [18 July 2020 20:03 UTC](https://discourse.nodered.org/t/http-request-node-gzip-encrypted-response/30300 "2020-07-18T20:03:42Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![Paul-Reed](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/paul-reed/32/66906_2.png) [@Paul-Reed](https://discourse.nodered.org/u/Paul-Reed)\
**Post date:** [18 July 2020 20:03 UTC](https://discourse.nodered.org/t/http-request-node-gzip-encrypted-response/30300/1 "2020-07-18T20:03:42Z")

</div>

Trying to download some statistical data from [https://c19downloads.azureedge.net/downloads/msoa\_data/MSOAs\_latest.json](https://c19downloads.azureedge.net/downloads/msoa_data/MSOAs_latest.json)

If I use that url in a browser, I get access to the full data, but when trying using the `http request` node I either get a lot of what appears to be encrypted data, or a message saying that the service is not available.

Are remote services able to block specific http requests (ie browsers only?).

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [18 July 2020 20:19 UTC](https://discourse.nodered.org/t/http-request-node-gzip-encrypted-response/30300/2 "2020-07-18T20:19:45Z")

</div>

If you use Devtools you can see what headers are used in the request & you can add those to the request.

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [18 July 2020 20:25 UTC](https://discourse.nodered.org/t/http-request-node-gzip-encrypted-response/30300/3 "2020-07-18T20:25:41Z")

</div>

> [@Paul-Reed](#):
>
> ie browsers only

Paul,  
I have seen similar behaviour already in the passed, due to a missing user-agent . See e.g. this [discussion](https://discourse.nodered.org/t/http-api-call-using-http-request-node-receives-access-denied-response/24976/3).

---

<div class="post-metadata">

**Author:** ![edje11](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/edje11/32/572_2.png) [@edje11](https://discourse.nodered.org/u/edje11)\
**Post date:** [18 July 2020 20:26 UTC](https://discourse.nodered.org/t/http-request-node-gzip-encrypted-response/30300/4 "2020-07-18T20:26:31Z")

</div>

Have a look at this topic, maybe you have to set the User-agent.

> [@Http API call using "http request" node receives "access denied" response](https://discourse.nodered.org/t/http-api-call-using-http-request-node-receives-access-denied-response/24976/3):
>
> Thank you, adding the header works: msg.url = "[https://api.weather.gov](https://api.weather.gov)"; msg.headers = {}; msg.headers['USER-AGENT'] = 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36'; return msg;

Edit: Bart was a couple of seconds faster.

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [18 July 2020 20:35 UTC](https://discourse.nodered.org/t/http-request-node-gzip-encrypted-response/30300/5 "2020-07-18T20:35:04Z")

</div>

> [@edje11](#):
>
> Bart was a couple of seconds faster

Yes my friend, speed means everything on this forum 🕶

---

<div class="post-metadata">

**Author:** ![Paul-Reed](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/paul-reed/32/66906_2.png) [@Paul-Reed](https://discourse.nodered.org/u/Paul-Reed)\
**Post date:** [18 July 2020 20:39 UTC](https://discourse.nodered.org/t/http-request-node-gzip-encrypted-response/30300/6 "2020-07-18T20:39:20Z")

</div>

I've copied the 'user-agent' entry from the browser Dev tools, and added them to a function node - feeding a http request node, but it's still returning unreadable data or denied.

```auto
msg.url = 'https://c19downloads.azureedge.net/downloads/msoa_data/MSOAs_latest.json';
msg.headers = {};
msg.headers['USER-AGENT'] = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36';
return msg;

```

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [18 July 2020 20:56 UTC](https://discourse.nodered.org/t/http-request-node-gzip-encrypted-response/30300/7 "2020-07-18T20:56:20Z")

</div>

I get this response when I navigate to the URL using Chrome:

## Our services aren't available right now

We're working to restore all services as soon as possible. Please check back soon.

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [18 July 2020 21:03 UTC](https://discourse.nodered.org/t/http-request-node-gzip-encrypted-response/30300/8 "2020-07-18T21:03:34Z")

</div>

And I get exactly the same result via a httprequest node:

```auto
[{"id":"55aff059.4d628","type":"http request","z":"be35bf3.aa6be4","name":"","method":"GET","ret":"txt","paytoqs":false,"url":"http://c19downloads.azureedge.net/downloads/msoa_data/MSOAs_latest.json","tls":"","persist":false,"proxy":"","authType":"","x":650,"y":560,"wires":[["181f6aa1.4a24c5"]]},{"id":"1d819cf8.726d23","type":"inject","z":"be35bf3.aa6be4","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"showConfirmation":false,"confirmationLabel":"","x":450,"y":560,"wires":[["55aff059.4d628"]]},{"id":"181f6aa1.4a24c5","type":"debug","z":"be35bf3.aa6be4","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":820,"y":560,"wires":[]}]

```

![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/2/b/2bcf36dd7220a686f46b61775a3cb1c945bd5bd5.png)

---

<div class="post-metadata">

**Author:** ![Paul-Reed](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/paul-reed/32/66906_2.png) [@Paul-Reed](https://discourse.nodered.org/u/Paul-Reed)\
**Post date:** [18 July 2020 21:06 UTC](https://discourse.nodered.org/t/http-request-node-gzip-encrypted-response/30300/9 "2020-07-18T21:06:12Z")

</div>

> [@BartButenaers](#):
>
> I get this response when I navigate to the URL

That's strange, using a uncached browser,I get the full data array of 6792 objects;

![reply](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/0/3/0307deec8d397ac31798abd175611f9735a7b7ac.jpeg)

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [18 July 2020 21:10 UTC](https://discourse.nodered.org/t/http-request-node-gzip-encrypted-response/30300/10 "2020-07-18T21:10:42Z")

</div>

> [@Paul-Reed](#):
>
> [http://c19downloads.azureedge.net/downloads/msoa\_data/MSOAs\_latest.json](http://c19downloads.azureedge.net/downloads/msoa_data/MSOAs_latest.json)

If I use https (instead of http) then I get a json file in chrome:

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/3/6/36adb733ac4985e9f50d77a0e35e544f94988ff3.png)

But I get some ugly stuff via the http-request node (which cannot be parsed to json):

![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/9/6/969306ad378d98744d302dff2e6a18ac503c032d.png)

---

<div class="post-metadata">

**Author:** ![Paul-Reed](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/paul-reed/32/66906_2.png) [@Paul-Reed](https://discourse.nodered.org/u/Paul-Reed)\
**Post date:** [18 July 2020 21:12 UTC](https://discourse.nodered.org/t/http-request-node-gzip-encrypted-response/30300/11 "2020-07-18T21:12:23Z")

</div>

Yes, that's exactly the same as what I get.  
(I am also now using the https url)

The request header is;

```auto
Accept	text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding	gzip, deflate, br
Accept-Language	en-GB,en;q=0.5
Connection	keep-alive
Host	c19downloads.azureedge.net
If-Modified-Since	Thu, 16 Jul 2020 14:21:40 GMT
If-None-Match	"0x8D829938E7B18BE"
TE	Trailers
Upgrade-Insecure-Requests	1
User-Agent	Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0

```

---

<div class="post-metadata">

**Author:** ![Paul-Reed](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/paul-reed/32/66906_2.png) [@Paul-Reed](https://discourse.nodered.org/u/Paul-Reed)\
**Post date:** [18 July 2020 21:30 UTC](https://discourse.nodered.org/t/http-request-node-gzip-encrypted-response/30300/12 "2020-07-18T21:30:30Z")

</div>

As the json response is large, is it possible that the data is compressed (which would explain the 'ugly stuff'?

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [18 July 2020 21:40 UTC](https://discourse.nodered.org/t/http-request-node-gzip-encrypted-response/30300/13 "2020-07-18T21:40:37Z")

</div>

> [@Paul-Reed](#):
>
> As the json response is large, is it possible that the data is compressed

I "think" you are right. This is what I get when I debug the http-request node:

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/0/a/0a89d485862d6ebd617eb9fe9c8c4b1edfbc9f8b.png)

So it gets a quite large body, which contains a zipped json.  
I have always been assuming that this kind of stuff happens behind the scenes automatically...

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [18 July 2020 21:44 UTC](https://discourse.nodered.org/t/http-request-node-gzip-encrypted-response/30300/14 "2020-07-18T21:44:15Z")

</div>

> [@BartButenaers](#):
>
> I have always been assuming that this kind of stuff happens behind the scenes automatically...

Found a small code snippet on [Stackoverflow](https://stackoverflow.com/a/19035763), that 'might' solve this:

```auto
request = require('request');
zlib = require('zlib');

request(url, {encoding: null}, function(err, response, body){
    if(response.headers['content-encoding'] == 'gzip'){
        zlib.gunzip(body, function(err, dezipped) {
            callback(dezipped.toString());
        });
    } else {
        callback(body);
    }
});

```

Will try it now ...

---

<div class="post-metadata">

**Author:** ![Paul-Reed](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/paul-reed/32/66906_2.png) [@Paul-Reed](https://discourse.nodered.org/u/Paul-Reed)\
**Post date:** [18 July 2020 21:50 UTC](https://discourse.nodered.org/t/http-request-node-gzip-encrypted-response/30300/15 "2020-07-18T21:50:02Z")

</div>

Adding a `node-red-contrib-gzip` node to the flow, plus a `json node`, and the output is correctly formed json!!

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/f/8/f885044839c4209c0fd55e69f0709e3b068b1328.png)

Thanks @BartButenaers

![output](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/a/7/a7ee9054c29603d9333c6618e04ccd69496ff665.jpeg)

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [18 July 2020 21:51 UTC](https://discourse.nodered.org/t/http-request-node-gzip-encrypted-response/30300/16 "2020-07-18T21:51:50Z")

</div>

Ah ok, that is faster compared to a pull-request to request a Node-RED core change 🤣  
Now you have beaten me in speed ...

---

<div class="post-metadata">

**Author:** ![Paul-Reed](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/paul-reed/32/66906_2.png) [@Paul-Reed](https://discourse.nodered.org/u/Paul-Reed)\
**Post date:** [18 July 2020 21:53 UTC](https://discourse.nodered.org/t/http-request-node-gzip-encrypted-response/30300/17 "2020-07-18T21:53:18Z")

</div>

> [@BartButenaers](#):
>
> Yes my friend, speed means everything on this forum 🕶

Very true!

---

<div class="post-metadata">

**Author:** ![Paul-Reed](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/paul-reed/32/66906_2.png) [@Paul-Reed](https://discourse.nodered.org/u/Paul-Reed)\
**Post date:** [1 August 2020 21:53 UTC](https://discourse.nodered.org/t/http-request-node-gzip-encrypted-response/30300/18 "2020-08-01T21:53:24Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.