# Http Request from Authenticated webpage

**URL:** <https://discourse.nodered.org/t/http-request-from-authenticated-webpage/74680>\
**Category:** General\
**Created:** [3 February 2023 21:55 UTC](https://discourse.nodered.org/t/http-request-from-authenticated-webpage/74680 "2023-02-03T21:55:08Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![jgreenmt](https://avatars.discourse-cdn.com/v4/letter/j/7feea3/32.png) [@jgreenmt](https://discourse.nodered.org/u/jgreenmt)\
**Post date:** [3 February 2023 21:55 UTC](https://discourse.nodered.org/t/http-request-from-authenticated-webpage/74680/1 "2023-02-03T21:55:08Z")

</div>

Hello,

The goal I am hoping to accomplish is reading data from a webpage that requires authentication to view.

For Example:  
When I login to my cable providers webpage, it displays how much my next bill will be, and when it is due. I want to be able to retrieve those values in Node-RED.

I have tried using an HTTP request node with the "use authentication" box checked and have verified I'm using the correct username and password.

I have also been able to use the "inspect" tool to find the tags of the information I want to retrieve.

The flow is `HTTP Request -> HTML -> Debug`

However I think the authentication part is what is causing me issues. I receive an empty payload when I specify any tags. If I read the output of the HTTP request directly it is the following:

```auto
<html style="height:100%"><head><META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"><meta name="format-detection" content="telephone=no"><meta name="viewport" content="initial-scale=1.0"><meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"></head><body style="margin:0px;height:100%"><iframe id="main-iframe" src="/_Incapsula_Resource?CWUDNSAI=42&xinfo=4-22405588-0%20NNNN%20RT%281675460190899%2017%29%20q%280%20-1%20-1%20-1%29%20r%280%20-1%29%20B10%2814%2c0%2c0%29%20U5&incident_id=7226000610184975022-113838336287313732&edet=10&cinfo=0e000000d66d&rpinfo=0&mth=GET" frameborder=0 width="100%" height="100%" marginheight="0px" marginwidth="0px">Request unsuccessful. Incapsula incident ID: 7226000610184975022-113838336287313732</iframe></body></html>

```

It appears this Incapsula Incident is more or less blocking my request and treating it as span or an attack of some kind. Just for added information, I have never seen additional security questions or anything on the login page (security questions, captcha tests, etc.).

Could anyone point me in the right direction? I can provide specific code if needed, but first I just wanted to see if I had overlooked something obvious, or if there was a security halt that prevents me from pursuing further.

---

<div class="post-metadata">

**Author:** ![jgreenmt](https://avatars.discourse-cdn.com/v4/letter/j/7feea3/32.png) [@jgreenmt](https://discourse.nodered.org/u/jgreenmt)\
**Post date:** [7 February 2023 20:24 UTC](https://discourse.nodered.org/t/http-request-from-authenticated-webpage/74680/2 "2023-02-07T20:24:00Z")

</div>

Can anyone point me in the right direction? If more information is needed I will be happy to provide.

---

<div class="post-metadata">

**Author:** ![bakman2](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bakman2/32/6207_2.png) [@bakman2](https://discourse.nodered.org/u/bakman2)\
**Post date:** [8 February 2023 06:49 UTC](https://discourse.nodered.org/t/http-request-from-authenticated-webpage/74680/3 "2023-02-08T06:49:01Z")

</div>

> [@jgreenmt](#):
>
> I have tried using an HTTP request node with the "use authentication" box checked and have verified I'm using the correct username and password.

The problem with help on this particular topic is that it requires authentication using credentials that only you own. The "authentication" settings in the the http request node are only valid if the website is using basic authentication. We don't know this.

There are many ways to implement security on websites and it all depends how they have set it up. Do they use basic auth, tokens, session cookies etc. The only way to figure this out is using the web inspector on the website you are trying to log in and follow the pages from login to the destination page and investigate what is passed between the pages.

---

<div class="post-metadata">

**Author:** ![jgreenmt](https://avatars.discourse-cdn.com/v4/letter/j/7feea3/32.png) [@jgreenmt](https://discourse.nodered.org/u/jgreenmt)\
**Post date:** [9 February 2023 17:03 UTC](https://discourse.nodered.org/t/http-request-from-authenticated-webpage/74680/4 "2023-02-09T17:03:52Z")

</div>

I appreciate the response and advice.

I have since decided to go a different direction using Selenium IDE to achieve this task. Everything works as expected, but now I want to call the script from Node-RED which has produced an error. I opened a new thread since it is technically a new issue.

Anyone who is curious, is welcome to head over and offer any insight/solutions.

> [@Working with Selenium IDE](https://discourse.nodered.org/t/working-with-selenium-ide/74949):
>
> Hello all, I have created a script using Selenium IDE that auto logins to a webpage and retrieves some information. Per the instructions I have downloaded the Selenium Side Runner (also known as Command Line Runner) tool that allows one to execute a project script from the command line. I am able to execute my script properly with no errors when using the command selenium-side-runner /path/to/file.side However, when I try to use an exec node to call this same command, it returns an error. F…

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [23 February 2023 17:04 UTC](https://discourse.nodered.org/t/http-request-from-authenticated-webpage/74680/5 "2023-02-23T17:04:13Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.