Trying to download some statistical data from https://c19downloads.azureedge.net/downloads/msoa_data/MSOAs_latest.json If I use that url in a browser, I get access to the full data, but when trying using the http request node I either get a lot of what appears to be encrypted data, or a message sa…

[image] Paul-Reed: As the json response is large, is it possible that the data is compressed I "think" you are right. This is what I get when I debug the http-request node: [image] So it gets a quite large body, which contains a zipped json. I have always been assuming that this kind of …

If you use Devtools you can see what headers are used in the request & you can add those to the request.

Have a look at this topic, maybe you have to set the User-agent. [image] Http API call using "http request" node receives "access denied" response FAQs Thank you, adding the header works: msg.url = " https://api.weather.gov "; msg.headers = {}; msg.headers['USER-AGENT'] =…

[image] edje11: Bart was a couple of seconds faster Yes my friend, speed means everything on this forum :sunglasses:

I've copied the 'user-agent' entry from the browser Dev tools, and added them to a function node - feeding a http request node, but it's still returning unreadable data or denied. msg.url = 'https://c19downloads.azureedge.net/downloads/msoa_data/MSOAs_latest.json'; msg.headers = {}; msg.headers['US…

I get this response when I navigate to the URL using Chrome: Our services aren't available right nowWe're working to restore all services as soon as possible. Please check back soon.

And I get exactly the same result via a httprequest node: [{"id":"55aff059.4d628","type":"http request","z":"be35bf3.aa6be4","name":"","method":"GET","ret":"txt","paytoqs":false,"url":"http://c19downloads.azureedge.net/downloads/msoa_data/MSOAs_latest.json","tls":"","persist":false,"proxy":"","auth…

[image] BartButenaers: I get this response when I navigate to the URL That's strange, using a uncached browser,I get the full data array of 6792 objects; [reply]

[image] Paul-Reed: http://c19downloads.azureedge.net/downloads/msoa_data/MSOAs_latest.json If I use https (instead of http) then I get a json file in chrome: [image] But I get some ugly stuff via the http-request node (which cannot be parsed to json): [image]

Yes, that's exactly the same as what I get. (I am also now using the https url) The request header is; Accept text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 Accept-Encoding gzip, deflate, br Accept-Language en-GB,en;q=0.5 Connection keep-alive Host c19downloads.azureed…

Node-RED

Node-RED Forum

Http request node - gzip encrypted response

General

BartButenaers 18 July 2020 20:25 3

Paul,
I have seen similar behaviour already in the passed, due to a missing user-agent . See e.g. this discussion.

Topic		Replies	Views
Basic url works fine in browser not in node-red HTTP request General	3	627	20 April 2020
Unreadable string from http request node General	9	340	10 September 2023
API call with gzip encoding General	5	177	26 June 2023
Get HTTP request for JSON data from BOM General	3	979	14 April 2021
Broken http request flow General http-request	3	265	25 October 2022

Copyright OpenJS Foundation and Node-RED contributors. All rights reserved. The OpenJS Foundation has registered trademarks and uses trademarks. For a list of trademarks of the OpenJS Foundation, please see our Trademark Policy and Trademark List. Trademarks and logos not indicated on the list of OpenJS Foundation trademarks are trademarks™ or registered® trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.

Http request node - gzip encrypted response

Related topics