statusCode 403 from https://www.volvocars.com/uk/support/downloads/maps/spa/europe

Hi!

I made a flow to check if there is a new version of maps from:

At the beginning, flow work as expected. I was able to get the payload from the page and check the new version. But one day, it "stopped working" and by that I mean, it always return a statusCode: 403.

Here is the code of my flow:

[{"id":"866fcc8c92f2cbbc","type":"inject","z":"c57baf29f28a2a71","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":380,"y":760,"wires":[["4edb78a02da14099"]]},{"id":"4edb78a02da14099","type":"http request","z":"c57baf29f28a2a71","name":"Get Sensus Navigation (2016) Map Download page","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://www.volvocars.com/nl/support/downloads/maps/spa/europa","tls":"","persist":true,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[{"keyType":"User-Agent","keyValue":"","valueType":"other","valueValue":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:107.0) Gecko/20100101 Firefox/107.0"},{"keyType":"Accept","keyValue":"","valueType":"other","valueValue":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"},{"keyType":"Accept-Encoding","keyValue":"","valueType":"other","valueValue":"gzip, deflate, br"},{"keyType":"Accept-Language","keyValue":"","valueType":"other","valueValue":"en-GB,en;q=0.5"}],"credentials":{},"x":470,"y":900,"wires":[["274dbbb7a7164ef4","8c4dce7d6b892fb0"]]},{"id":"8c4dce7d6b892fb0","type":"debug","z":"c57baf29f28a2a71","name":"debug 4","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":1020,"y":720,"wires":[]}]

I don't know exactly what am I missing to make this flow work again. Can someone provide a bit of help on what is going on? Any help is really appreciated.

Regards,
acaminiti

Have you tried entering the url in a browser directly to see what happens?

An HTTP 403 response code means that a client is forbidden from accessing a valid URL . The server understands the request, but it can't fulfill the request because of client-side issues. The caller isn't authorized to access an API that's using an API Gateway Lambda authorizer.

They have probably changed something. You could try asking them for help.

@zenofmud don't think that Volvo would answer to a random guy that just wants to scrap information from their site because he is lazy. I did tried the link on my web-browser and yes I can see the page.

My guess is that I'm probably missing web-browser agent info and some other information that is exchange between the web-browser and the server. I did tried to use Firefox and Chrome web developers tools to gather that info, but I'm not sure what I should be looking for and how to set it in the node in order to generate a legit request.

Regards.

There's another page (VOLVO USA) which lists the version. Not sure if it's always the same compared to EU Europe though.

Don't judge my code please :nerd_face: but the HTML of this site is ugly, so I did an ugly string search too.

[{"id":"2b380e8543f43e3d","type":"tab","label":"Flow 3","disabled":false,"info":"","env":[]},{"id":"3d3256309491a55b","type":"http request","z":"2b380e8543f43e3d","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://volvo.custhelp.com/app/answers/detail/a_id/9378/~/download-the-latest-maps","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[],"x":330,"y":80,"wires":[["0e2d32d0acf15b25"]]},{"id":"d8f9610e663cd64c","type":"inject","z":"2b380e8543f43e3d","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":140,"y":80,"wires":[["3d3256309491a55b"]]},{"id":"85dc05ae0c0c01bd","type":"debug","z":"2b380e8543f43e3d","name":"debug 103","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":750,"y":80,"wires":[]},{"id":"0e2d32d0acf15b25","type":"function","z":"2b380e8543f43e3d","name":"get map version","func":"let index = msg.payload.indexOf(\"version is\");\nlet tmp_string = \"\";\nif(index > -1 ){\n    \n    tmp_string = msg.payload.substring(index, index + 60)\n    index = tmp_string.indexOf(\">\");\n    tmp_string = tmp_string.substring(index+1);\n    index = tmp_string.indexOf(\"&\");\n    tmp_string = tmp_string.substring(0,index);\n    \n}\nmsg.payload = tmp_string;\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":540,"y":80,"wires":[["85dc05ae0c0c01bd"]]}]

One day we'll get a VOLVO too, I hope.

Thanks @rko. The code indeed works. This other website looks a bit shady as it is not volvo.com
Let me check it, and thanks. Volvo is leased. :stuck_out_tongue:

Dear @rko indeed code worked. But now it just happen what i suspected it would happen by not using the official website. Versions between websites are out of sync. Seem that I might need to find a more reliable way to scrape Volvo's website.

Have you tried including the Cookies in the request headers that may be needed for the Authentication ?

Example Flow

[{"id":"866fcc8c92f2cbbc","type":"inject","z":"54efb553244c241f","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":220,"y":2300,"wires":[["c933ee818f640fab"]]},{"id":"8c4dce7d6b892fb0","type":"debug","z":"54efb553244c241f","name":"debug 4","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":940,"y":2300,"wires":[]},{"id":"8d6f6e8943dc0ba4","type":"http request","z":"54efb553244c241f","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://www.volvocars.com/nl/support/downloads/maps/spa/europa","tls":"","persist":true,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[],"x":590,"y":2300,"wires":[["069e4e4f83775d06","e990027560a2ca03"]]},{"id":"c933ee818f640fab","type":"function","z":"54efb553244c241f","name":"headers","func":"msg.headers = {\n\n    \"Accept\": \"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8\",\n    // \"Accept-Encoding\": \"gzip, deflate, br\",\n    \"Accept-Language\": \"en-US,en;q=0.5\",\n\n    \"Connection\": \"keep-alive\",\n    \"Cookie\": \"OptanonConsent=isGpcEnabled=0&datestamp=Mon+Jan+23+2023+22%3A52%3A42+GMT%2B0200+(Eastern+European+Standard+Time)&version=202209.1.0&isIABGlobal=false&hosts=&consentId=81ba6176-5bd2-4ed7-80aa-6337db2022fe&interactionCount=1&landingPath=NotLandingPage&groups=1%3A1%2C2%3A0%2C3%3A0%2C4%3A0%2C9%3A0; OptanonAlertBoxClosed=2023-01-23T20:52:42.215Z\",\n    \"Host\": \"www.volvocars.com\",\n\n    \"Upgrade-Insecure-Requests\": \"1\",\n    \"User-Agent\": \"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0\"\n}\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":420,"y":2300,"wires":[["8d6f6e8943dc0ba4"]]},{"id":"069e4e4f83775d06","type":"html","z":"54efb553244c241f","name":"","property":"payload","outproperty":"payload","tag":"tbody td","ret":"text","as":"single","x":780,"y":2300,"wires":[["8c4dce7d6b892fb0"]]},{"id":"e990027560a2ca03","type":"debug","z":"54efb553244c241f","name":"debug 24","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":710,"y":2240,"wires":[]}]

ps. the cookie from what i see in my Firefox dev tools expire in a year so you may need to devise a way to dynamically update them

2 Likes

Thanks @UnborN this is exactly what I was looking for. Indeed the cookie and some others were the missing pieces. I did tried only adding Accept, Accept-Language, Encoding and Host. But as there are not so many examples on how to properly populate the headers in the http-request is was kind of difficult for me to figure it out. With the example you provided, I even managed to attach all the header elements into the same http-request node. and it is working like a charm!

Full flow here:

[{"id":"678cfef18850b1fc","type":"html","z":"c57baf29f28a2a71","name":"Get Latest version","property":"payload","outproperty":"payload","tag":"#root > div > div > main > div > div.a.bs.bt.d > div:nth-child(2)","ret":"html","as":"multi","x":1050,"y":940,"wires":[["19492f0e1e2780bb"]]},{"id":"45552477e59279f2","type":"string","z":"c57baf29f28a2a71","name":"Evaluate string for: SW details","methods":[{"name":"contains","params":[{"type":"str","value":"Software details"}]}],"prop":"payload","propout":"payload","object":"msg","objectout":"msg","x":270,"y":1040,"wires":[["33211a6121eb1a19"]]},{"id":"063a7d68e2cc85d8","type":"string","z":"c57baf29f28a2a71","name":"Evaluate string for: Version","methods":[{"name":"contains","params":[{"type":"str","value":"Version"}]}],"prop":"payload","propout":"payload","object":"msg","objectout":"msg","x":260,"y":1080,"wires":[["2dc4e54f3db812c2"]]},{"id":"a38a16af9d39b3f7","type":"BooleanLogicUltimate","z":"c57baf29f28a2a71","name":"AND","filtertrue":"both","persist":true,"sInitializeWith":"WaitForPayload","triggertopic":"trigger","outputtriggeredby":"all","inputCount":"2","topic":"valid_html","restrictinputevaluation":true,"delayEvaluation":"100","x":740,"y":1060,"wires":[["21b53920806ec6d6"],[],[]]},{"id":"33211a6121eb1a19","type":"change","z":"c57baf29f28a2a71","name":"Set topic","rules":[{"t":"set","p":"topic","pt":"msg","to":"software_details","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":520,"y":1040,"wires":[["a38a16af9d39b3f7"]]},{"id":"2dc4e54f3db812c2","type":"change","z":"c57baf29f28a2a71","name":"Set topic","rules":[{"t":"set","p":"topic","pt":"msg","to":"version","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":520,"y":1080,"wires":[["a38a16af9d39b3f7"]]},{"id":"6864184213c8949c","type":"InterruptFlowUltimate","z":"c57baf29f28a2a71","name":"Interrupt Flow","triggertopic":"valid_html","initializewith":"1","autoToggle":"0","payloadPropName":"payload","x":1170,"y":1100,"wires":[["83064bf95bf60f42"]]},{"id":"1df25504dbb36751","type":"string","z":"c57baf29f28a2a71","name":"Find Version Number","methods":[{"name":"between","params":[{"type":"str","value":"Version</td><td>"},{"type":"str","value":"</td></tr><tr><td>Planned update"}]},{"name":"trim","params":[]},{"name":"toInteger","params":[]}],"prop":"payload","propout":"payload","object":"msg","objectout":"msg","x":240,"y":1120,"wires":[["6864184213c8949c"]]},{"id":"21b53920806ec6d6","type":"switch","z":"c57baf29f28a2a71","name":"","property":"payload","propertyType":"msg","rules":[{"t":"false"},{"t":"true"}],"checkall":"true","repair":false,"outputs":2,"x":950,"y":1060,"wires":[["a3eb8ba0d5e8e467","8cc24db782522aeb"],["6864184213c8949c"]]},{"id":"1231078d6c610abc","type":"function","z":"c57baf29f28a2a71","name":"Save to Global Current Version","func":"global.set(\"sensus_installed_version\", msg.payload, \"file\")\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":510,"y":1200,"wires":[["e3fa29a2e147bdcb","8cc24db782522aeb"]]},{"id":"83064bf95bf60f42","type":"switch","z":"c57baf29f28a2a71","name":"Compare versions","property":"payload","propertyType":"msg","rules":[{"t":"neq","v":"#:(file)::sensus_installed_version","vt":"global"},{"t":"eq","v":"payload","vt":"msg"}],"checkall":"true","repair":false,"outputs":2,"x":230,"y":1200,"wires":[["1231078d6c610abc"],[]]},{"id":"274dbbb7a7164ef4","type":"switch","z":"c57baf29f28a2a71","name":"Check Status Code","property":"statusCode","propertyType":"msg","rules":[{"t":"neq","v":"200","vt":"num"},{"t":"eq","v":"200","vt":"num"}],"checkall":"false","repair":false,"outputs":2,"x":810,"y":900,"wires":[["4cf0f3cd905083f9","8cc24db782522aeb"],["678cfef18850b1fc"]]},{"id":"aab794143527c23c","type":"http request","z":"c57baf29f28a2a71","name":"Get Sensus Navigation (2016) Map Download page","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://www.volvocars.com/uk/support/downloads/maps/spa/europe","tls":"","persist":true,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[{"keyType":"other","keyValue":"Accept","valueType":"other","valueValue":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"},{"keyType":"other","keyValue":"Accept-Language","valueType":"other","valueValue":"en-US,en;q=0.5"},{"keyType":"other","keyValue":"Connection","valueType":"other","valueValue":"keep-alive"},{"keyType":"other","keyValue":"Cookie","valueType":"other","valueValue":"OptanonConsent=isGpcEnabled=0&datestamp=Tue+Jan+24+2023+22%3A06%3A12+GMT%2B0100+(Central+European+Standard+Time)&version=202212.1.0&isIABGlobal=false&hosts=&consentId=3693cd21-8286-4586-ba69-962a45490fc9&interactionCount=1&landingPath=https%3A%2F%2Fwww.volvocars.com%2Fuk%2Fsupport%2Fdownloads%2Fmaps%2Fspa%2Feurope&groups=1%3A1%2C2%3A0%2C3%3A0%2C4%3A0%2C9%3A0\""},{"keyType":"other","keyValue":"Host","valueType":"other","valueValue":"www.volvocars.com"},{"keyType":"other","keyValue":"Upgrade-Insecure-Requests","valueType":"other","valueValue":"1"},{"keyType":"other","keyValue":"User-Agent","valueType":"other","valueValue":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0"}],"x":470,"y":900,"wires":[["274dbbb7a7164ef4"]]},{"id":"8cc24db782522aeb","type":"debug","z":"c57baf29f28a2a71","name":"debug 104","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":910,"y":760,"wires":[]},{"id":"5585492ec91c9268","type":"inject","z":"c57baf29f28a2a71","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":360,"y":720,"wires":[["aab794143527c23c"]]},{"id":"83c96c84fa59db55","type":"junction","z":"c57baf29f28a2a71","x":140,"y":1040,"wires":[[]]},{"id":"19492f0e1e2780bb","type":"junction","z":"c57baf29f28a2a71","x":80,"y":1020,"wires":[["45552477e59279f2","063a7d68e2cc85d8","1df25504dbb36751"]]}]

Really appreciate your help and support.

2 Likes