Http Request from Authenticated webpage

General
1

Hello,

The goal I am hoping to accomplish is reading data from a webpage that requires authentication to view.

For Example:
When I login to my cable providers webpage, it displays how much my next bill will be, and when it is due. I want to be able to retrieve those values in Node-RED.

I have tried using an HTTP request node with the "use authentication" box checked and have verified I'm using the correct username and password.

I have also been able to use the "inspect" tool to find the tags of the information I want to retrieve.

The flow is HTTP Request -> HTML -> Debug

However I think the authentication part is what is causing me issues. I receive an empty payload when I specify any tags. If I read the output of the HTTP request directly it is the following:

<html style="height:100%"><head><META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"><meta name="format-detection" content="telephone=no"><meta name="viewport" content="initial-scale=1.0"><meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"></head><body style="margin:0px;height:100%"><iframe id="main-iframe" src="/_Incapsula_Resource?CWUDNSAI=42&xinfo=4-22405588-0%20NNNN%20RT%281675460190899%2017%29%20q%280%20-1%20-1%20-1%29%20r%280%20-1%29%20B10%2814%2c0%2c0%29%20U5&incident_id=7226000610184975022-113838336287313732&edet=10&cinfo=0e000000d66d&rpinfo=0&mth=GET" frameborder=0 width="100%" height="100%" marginheight="0px" marginwidth="0px">Request unsuccessful. Incapsula incident ID: 7226000610184975022-113838336287313732</iframe></body></html>

It appears this Incapsula Incident is more or less blocking my request and treating it as span or an attack of some kind. Just for added information, I have never seen additional security questions or anything on the login page (security questions, captcha tests, etc.).

Could anyone point me in the right direction? I can provide specific code if needed, but first I just wanted to see if I had overlooked something obvious, or if there was a security halt that prevents me from pursuing further.

2

Can anyone point me in the right direction? If more information is needed I will be happy to provide.

3

The problem with help on this particular topic is that it requires authentication using credentials that only you own. The "authentication" settings in the the http request node are only valid if the website is using basic authentication. We don't know this.

There are many ways to implement security on websites and it all depends how they have set it up. Do they use basic auth, tokens, session cookies etc. The only way to figure this out is using the web inspector on the website you are trying to log in and follow the pages from login to the destination page and investigate what is passed between the pages.

4

I appreciate the response and advice.

I have since decided to go a different direction using Selenium IDE to achieve this task. Everything works as expected, but now I want to call the script from Node-RED which has produced an error. I opened a new thread since it is technically a new issue.

Anyone who is curious, is welcome to head over and offer any insight/solutions.

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.