Grab details from password protected website

General
1

Hi,

I have a DataLogger for my Solar System and I want to extract values from the internal Webserver of the solar heating system.

It is a system from ta.co.at (UVR1611) and as Datalogger I am using the CMI from the same company.

This CMI has an integrated API and I can get most of the needed information via the API.
For the "not accessable" values I can create a so called schema, which I can access via a webbrowser + user/pass.

It is a website, where you get the Popup from the browser to enter the username/password.

If I want to directly access this Schema, I need to go to "http://192.168.1.19/schema.html#1". I am getting a popup for User/Password and I only need to enter "gast" as user. Then I am able to see the website.

I tried the following:

[{"id":"da5794f7.b89e78","type":"http request","z":"d2043ed8.8e731","name":"","method":"GET","ret":"txt","paytoqs":false,"url":"http://192.168.1.19/schema.html#1","tls":"","proxy":"","authType":"basic","x":430,"y":3040,"wires":[["79ecd2cf.2ebacc"]]},{"id":"44fe52c7.a9fb2c","type":"inject","z":"d2043ed8.8e731","name":"","topic":"","payload":"","payloadType":"date","repeat":"60","crontab":"","once":true,"onceDelay":"60","x":210,"y":3040,"wires":[["da5794f7.b89e78"]]},{"id":"79ecd2cf.2ebacc","type":"debug","z":"d2043ed8.8e731","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":670,"y":3040,"wires":[]}]

It seems that I am getting connected, because with a wrong username I am getting back a 404.
With the correct username I am getting back a website, but unfortunatley without values. It seems that the website is displaying the values via javascript which seems not to be handeled by the http request node.

This is what I am getting back:

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="target-densitydpi=low-dpi" />
    <meta http-equiv="X-UA-Compatible" content="IE=100" >
    <title>Schema</title>
    <link rel="stylesheet" type="text/css" href="style/webi/jquery-ui-1.10.0.custom.css" />
    <link rel="stylesheet" type="text/css" href="style/schema.css" />
    <script type="text/javascript" src="js/jquery-1.9.0.js" ></script>
    <script type="text/javascript" src="js/jquery-ui.js" ></script>
    <script type="text/javascript" src="js/jquery.ui.touch-punch.js" ></script>
    <script type="text/javascript" src="js/scheme.js" ></script>
	<script type="text/javascript" src="js/schemabrand.js" ></script>
	<script type="text/javascript">
	var os = "";
	var refreshtime = 0;
	var aktiv;
	var titel = "SCHEMA";
	var image = "";
	var notFound = "Schema nicht gefunden!";
	var howtotxt = "Wie erstelle ich ein Schema?";
	var	howto = "";
	</script>
  </head>
  <body>
		<div style="position:relative;display:inline-block;">
  		<div id="prop"></div>
  		<div id="values"></div>
		<div id="schema"></div>
		<div id="changer">CHANGE</div>
  	</div>
  	<span id='s_OK' class='lang'>OK</span>
    <span id='s_Abbrechen' class='lang'>Abbrechen</span>
  </body>
</html>

Any ideas?

Br,
JOhannes

2

This type of authentication is called Basic Auth, and is supported out of the box by the http request core node. Using that node, set the auth type to basic, enter gast in the username field, save and try to request it. From the flow that you posted it appears you are doing that. Are you 100% sure that the URL you are requesting is correct?

3

Hi,

Yes, I am pretty sure the URL is correct, and also authentication is working.

If I open the page via chrome and then "right click" on the page to check the source, then it looks identically:

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="target-densitydpi=low-dpi" />
    <meta http-equiv="X-UA-Compatible" content="IE=100" >
    <title>Schema</title>
    <link rel="stylesheet" type="text/css" href="style/webi/jquery-ui-1.10.0.custom.css" />
    <link rel="stylesheet" type="text/css" href="style/schema.css" />
    <script type="text/javascript" src="js/jquery-1.9.0.js" ></script>
    <script type="text/javascript" src="js/jquery-ui.js" ></script>
    <script type="text/javascript" src="js/jquery.ui.touch-punch.js" ></script>
    <script type="text/javascript" src="js/scheme.js" ></script>
		<script type="text/javascript">
		var os = "";
		var refreshtime = 0;
		var aktiv;
		var titel = "SCHEMA";
		var image = "";
		var notFound = "Scheme not found!";
		var howtotxt = "How to create an online scheme?";
		var howto =  "<div id='help'><a href='http://help.ta.co.at/EN/TA_Designer/_step_by_step__instructions.htm' target='_BLANK'>"
					+"<h3><img width='16' height='16' alt='help' src='images/help.png'>"+howtotxt+"</h3></a></div>";
		</script>
  </head>
  <body>
		<div style="position:relative;display:inline-block;">
  		<div id="prop"></div>
  		<div id="values"></div>
		<div id="schema"></div>
		<div id="changer">CHANGE</div>
  	</div>
  	<span id='s_OK' class='lang'>OK</span>
	<span id='s_Abbrechen' class='lang'>Cancel</span>
  </body>
</html>

I think the values are inserted via the javascript and this is maybe not working with the http request node?`

This is when analyising the website via chrome developer mode:
image

4

Yes it is. Open the page while looking at the network tab of the developer tools. That should show which resources are loaded with the actual data.

5

image
image557Ă—518 13.9 KB

6

This are the values I am looking for:

image
image552Ă—524 13.3 KB

7

well guess that's your answer, the link for 1.cgi with 1 URL parameter called _ (used to prevent caching) that is the timestamp you can get from an inject node. So you set it to use the payload as URL parameters, then previously in a change node set the payload to {"_": <timestamp from inject>}, for example inject set to mode timestamp, change node with "move" msg.payload to msg.payload._.

2 Likes