Scraping Voltaware meter data?

[Mart]

I want to poll a Voltaware monitor that’s measuring power consumption in my house. I can use a browser to view the data direct from the monitor (image below). I thought I could “just” use an “http request” node in Node-RED and parse the html UTF-8 response. However, the payload is gobbledegook - example below? What am I doing wrong?

��&�b�meterdata.html�VMs�6��W �E� �t�C’9�q�ine;�LO�%�����_�)%�d�b���[`�vW�W������ZTdt~6���L�oB�4柝&X�Gqo���AB?w�� �����% ��ߒ���Y�T����DHg -CW��*+q�$���PV�� �_���4f���a�P�S� �� ��R 5{��1�O3Ԅ�ϟ��U�0ie�G�%-0T�|���<K�B�2���_��P�&��eI��G6����a�UM"xYe�닯!�'�n�8��K` �q�%����`x6w�<��\g��2KjX�,~�η�+�R(�@>,�kl9S&�*�K���!e�4���܉)�D��h�AR�Xv���� |֮t��q�[��ʀ����.�ȝ͒�zIJ��2U��6˔]�?ٕ7��w{Hؼ�� qO����f�j�3R��ԫE� p�J+���I~�M�pć�%>^�:��r�=i���V{��������a��Sܤ�~b%Zy�t�/�}/� h�} ���V^�̃�/y�8Q��m�����B�Dž�w�k��-Z�2��%\�m�A<��Y+^y` �K�[����_�;�e�eҧ�k��W��u���k�e�o3�L���ߒ6��;K��:���B�[u��fT�#���.u�al��i9���4x��N�>��nt�x%7�`4�/�|�˻!���7Kqm�/�b4=g���0�2����we#�a��!�KIQ�[�/C��v�F"
Q�Wj�B)q����=r;�����;F<�!����P,�a�f,’��.�k޽Ͻ�1w��5�C^���݀�e�s��ņ�3

[E1cid]

What headers does it send, it is probably compressed.
this post may help

Http request node - gzip encrypted response

Adding a node-red-contrib-gzip node to the flow, plus a json node, and the output is correctly formed json!! [image] Thanks @BartButenaers [output]

[Mart]

Brilliant - found the gzip node, added it, changed the http request node to return a binary buffer ... and the gzip node produced recognizable html.

I have a debug node connected - but it only shows the first 28 lines of the web page...

I've added a write file node and it outputs the full html for the page OK.

The html creates a table to show data but there's no data for me to extract in the html itself - looks like the page uses some javascript routines to populate the data e.g.

<script src="jquery.js"></script>
<script src="jquery.mobile.js"></script>
<script src="common.js"></script>
<script src="meterdata.js"></script>

Not sure where to go next - do I need to run the javascript somehow to read the data?

[Mart]

Had a rummage around in the javascript and there was another URL that returns the data in json format.

I can now poll the Voltaware monitor and get “live” power consumption data - just what I wanted!