Http request to machine's screen mirror

[lastheim]

Hi,

Im trying to get values from my boiler via its web screen mirror/GUI

When I send the GET request directly to main resource the output is non-readable and not the html doc I was expecting

How should I configure the request to get proper html and to be able to access a specific value from the screen mirror?

[{"id":"d88dd470.0ac7b8","type":"inject","z":"a9b2455f.5eebd8","name":"make request","props":[{"p":"payload","v":"","vt":"date"},{"p":"topic","v":"","vt":"string"}],"repeat":"","crontab":"","once":false,"topic":"","payload":"","payloadType":"date","x":590,"y":500,"wires":[["874a3d4e.9b666"]]},{"id":"874a3d4e.9b666","type":"http request","z":"a9b2455f.5eebd8","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"http://192.168.1.166:80/main.html","tls":"","persist":true,"proxy":"","authType":"","x":890,"y":500,"wires":[["7403c68f.21d7c8"]]},{"id":"7403c68f.21d7c8","type":"debug","z":"a9b2455f.5eebd8","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":1190,"y":500,"wires":[]}]

Output is:

��[}s�6����%+����2�q�M'o;�>��< IH(�G��][�����t_�v�)�Ig:s3Gv��b������?G��oO/����Z�U||$����y�1&�ϲ$;�ƣ��%��o8[�I&�0� �5����a4�\� �a3���Qf<���B��'���Y�3۹Ϙ,2a�Y"�0��d8$}�nL�?R[Ԕ5+��S
����I�3)�X�C$!
�wA��R���i{��,<�����,qlV
$;�ܾ�}b����ӌt^������'�볋��>�$=?{����Ň7�^Fq������՛o��L����${����A~?o(D����ν��n��B.�|�.�Y������ ϝ�,���9���*�a�ٵ�10X%N�aP��\-�i s���D��?2��tp�Ӧr���G���S^�+���O�%?_�[���P�g��:�x�j�uq����#�Xc��X 7�v��������?_��M��=����r����E��r +e.�~50�\��U�H��OlՈE�����Ԝ��T}�zۚ�f !O��Zq�{�
�����3s����k�l�;�'�2���~���g��ș���t���R� ˂�Y1����]p��\��!Xf�0�e�Ȓ�P"��j�֪�� c7A�#���Ԣ�ZҌU_)z����ҳ۔��@�e�����a��yd�����~}}��9�_�Ii>�־/�8�"��l�+��Ά�9�xO�ps��XEz�)���;�BQ+�.D�����gw/#���g7�'���l��|j��@ViJ~=� h8����M��@2N$׊�(@�NMnj,���*�U�3����4��� 0j��b��6h.�bv�dL�e�99�$�34Ɏ��.GWnVĬb�d��2�Q����15�p#���|�e��a*!$2�ϋ,)Dt��IF�B�r[>���>OWؤ
5Ŧa�2
��cT&�?�\�$a+����Z��[ۑo`F�y[5J�!���19ܙ�a�Dw�v�...

from webpage:

[Colin]

Is the node red server running on the same PC as the browser screenshot?

Are you sure that is exactly the same url that you use in the browser? (http://192.168.1.166:80/main.html)

Set the debug node to output Complete Message and see what else you see.

[lastheim]

The node red server is running on an raspberry pi, docker image setup according to IOTstack

Yes, it's the url Im using, I've tested some addresses:

http://192.168.1.166/main.htm (as described in the boilers manual)
http://192.168.1.166/main.html
http://192.168.1.166:80/main.html

all gives the same result as described

Output of complete message:

[Colin]

@lastheim

The node red server is running on an raspberry pi, docker image setup according to IOTstack

What happens if you use curl or wget to get the page? Try it on your PC and if that works then from a terminal inside the docker image.

[lastheim]

Output from curl in cmd of Windows PC, seems like the same output as from NR debug node

C:\WINDOWS\System32>curl -v "http://192.168.1.166/main.htm"

• Trying 192.168.1.166...
• TCP_NODELAY set
• Connected to 192.168.1.166 (192.168.1.166) port 80 (#0)

GET /main.htm HTTP/1.1
Host: 192.168.1.166
User-Agent: curl/7.55.1
Accept: /

< HTTP/1.1 200 OK
< Connection: close
< Content-Length: 005184
< Content-Type: text/html; charset=utf-8
< Access-Control-Allow-Origin: *
< Content-Encoding: gzip
<
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: " to save to a file.

• Failed writing body (0 != 842)
• Closing connection 0

[Colin]

OK, so it is not node-red. I suspect that it is that the device looks at the request headers and is doing different things dependent on the headers. You could try setting the user agent to something like
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:54.0) Gecko/20100101 Firefox/69.0
Others may have better suggestions.

[dceejay]

@lastheim

Content-Encoding: gzip

Well it says the output is zipped - so indeed as curl suggests ry saving it to a file then unzipping it... to see. (but yes faking the headers as @colin suggests may work better in the long term)

[lastheim]

I don't know what to make of this but I tried your, @Colin and @dceejay, suggestions combined:

C:\WINDOWS\System32>curl -v -A "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:54.0) Gecko/20100101 Firefox/69.0" http://192.168.1.166/main.html -o output.gzip

The content of the gzip is of doctype html but doesn't contain all values that Im looking for but I guess it's kind of the first layer of the frontend, there are tags for remoteScreens and adminContent which are the tab of the GUI.

Is it possible to send a command, like "access remoteScreens", through curl?

[Colin]

If it is still gzipped then the user agent probably isn't making any difference. It must be using something else.

[Steve-Mcl]

Open your browser Dev tools, select the network tab, refresh your browser and look at the requests made. It is quite possible the values you are looking for are in some Json request.

Alternatively, I have seen some of these systems where the remote view is just a bitmap - you might see this in the network requests.

Also, I have seen some of these systems where the remote view is some kind of vnc. If that is the case good luck getting the values or a screen capture.

[Colin]

Google has just informed me that browsers are capable of interpreting gzipped html, so it may be that it is always gzipped, but when the browser fetches it it automatically unzips it.

[lastheim]

So I've had some progress using the node-red-contrib-gzip node. Now Im able to request the attributes of an element (!?) called "remote"

But now Im stuck since I need to trigger "onClick: "_switch(0);" to access the sensor data that Im after.

payload: object
id: "remote"
onclick: "_switch(0);"

This is what the html section looks like:

<div id=head class=noselect style=left:10px;top:10;width:480px;height:30px;color:#000><nav><ul id=menu name=menu><li><span id=remote onclick=_switch(0);>remote</span><li><span id=admin onclick=_switch(1);>admin</span></ul></nav></div>

[Steve-Mcl]

@lastheim

But now Im stuck since I need to trigger "onClick: "_switch(0);" to access the sensor data that Im after.

Http request only retrieves data - it is not an active environment/browser. I.e. it cannot operate scrips.

Did you look in devtools network tab as I indicated before?

[afelix]

I hope Steve won't mind if I come with a few suggestions as well, so I'll be adding a bit of background to what he is asking you to do, which should help in figuring it out.

The browser's "development tools" allow you to figure out a lot of information from a webpage. There's a bunch of tabs there with different functions and possibilities. The default tab just shows you an interactive view of the page's source. The debugger tab lets you view the direct source code of all linked files, and is what I often use if I'm diving deeper into a page. The network tab logs all requests from the moment you load a page, for other resources that are being loaded. Everything from images to additional javascript files, to requests done from the webpage: to sensor data for example.

If you use the http-request node in the browser, it gives the output of the contents on that exact URL only. As it is plain text (or as in your case gzipped plain text/html), you can't interact with any of the contents, like javascript actions on the page. If the sensor data was inserted into the HTML directly, you could use this to extract the data from it.

Your options here are as follows: check the network tab for other resources being loaded, which could be javascript or perhaps json files. Either find the direct files that are being loaded, or find the javascript segment with a _switch(...) function defined, where you can work out (from perhaps the debugger tab) what it does, and where it gets the data from. I hope this helps you around!

Edit1: looking at your initial post with javascript screenshot, and the manual of the device, one thing that might be the case is something I've seen before with devices of the brand Omron: the web page might just be there to forward places you tap on the screen, and tells the device to do something with that info. If that's the case, that's incredible not useful when reverse engineering the data, but it's something to be aware of that it's a possibility too. Looks like you might be able to get info out though, but it might be more work than you had expected at first. The second option if that's needed is take the android app they offer (My Uplink app), and decompile the APK file and work your way through the Java code, as it apppears to have a way to log in/connect, and can list the raw numbers in the app, suggesting there's an internal API that they have available.

[lastheim]

Thanks guys! Im now able to access to the boilers data.

Should have focused more on the dev tools as @Steve-Mcl pointed out from the beginning.

By looking at the requests executed by clicking in the interface I got an URL: http://192.168.1.166/vars/glob;menu;1;2;82;87;83

...where the different pages contained boiler variables in a string, e.g. 0|0|70|1|2|1|0|210|1|0|0|0|1|1|