Extracting HTML data from table unable to retrieve more than headers

General
1

I am extracting data from an html file https://www.vcwatershed.net/fws/reservoir.html. I am successful in getting the "headings" but I am unable to retrieve the rest of the "tr" rows from this file. The html file and the specific "tr" I am after looks like the below:

<html>
 <head>...</head>
  <body>
   <body">
       <a name="top"></a>
       <h1>VCWPD Area Reservoir Report</h1>
       <table border="1" id="weather_table">
         <tbody>
           <tr class="header">...</tr>
           <tr class="row0">...</tr>
           <tr class="row1">
             <td>691</td>
             <td>Lake Casitas Dam level</td>
             <td>11/29/2019 09:29</td>
             <td align="Right">567.00</td>
             <td align="Right">238,000</td>
             <td align="Right">500.34</td>
             <td align="Right">98,037</td>
             <td align="Right">---</td>
             <td align="Right">0</td>
             <td align="Right">-66.66</td>
             <td align="Right">41.2%</td>
            </tr>
           <tr class="row0">...</tr>
           <tr class="row1">...</tr>
           <tr class="row0">...</tr>
           <tr class="row1">...</tr>
     </tbody>
    </table>
   </body">
  </body>
</html>

I am totally new to this and have used the following selector's:

#weather_table > tbody > tr:nth-child(3)

#weather_table > tbody > tr

Whichever css selector I try I only get the headers (first tr). Any help would be appreciated

EDIT

This works fine when above html is copied into Node-red but the css selector does not work when using the actual html website. Refer below flow:

[{"id":"92fa50d7.6f87a","type":"html","z":"e36bab88.137a18","name":"","property":"payload","outproperty":"payload","tag":"#weather_table > tbody > tr:nth-child(3)","ret":"html","as":"single","x":2780,"y":500,"wires":[["c90d8845.e8d708"]]},{"id":"c90d8845.e8d708","type":"debug","z":"e36bab88.137a18","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":3070,"y":500,"wires":[]},{"id":"4a59a024.12515","type":"inject","z":"e36bab88.137a18","name":"get data","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"x":2220,"y":500,"wires":[["3fc08d79.ea08b2"]]},{"id":"3fc08d79.ea08b2","type":"https-node","z":"e36bab88.137a18","name":"Request reservoir table","method":"GET","ret":"txt","url":"https://www.vcwatershed.net/fws/reservoir.html","authorized":false,"agent":true,"x":2430,"y":500,"wires":[["92fa50d7.6f87a"]]}]
2

My simple test case seems to work...

[{"id":"a9018ec0.88f5d","type":"inject","z":"ea593186.4918d","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":100,"y":720,"wires":[["6fd6828d.f40d5c"]]},{"id":"6fd6828d.f40d5c","type":"template","z":"ea593186.4918d","name":"","field":"payload","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"<html>\n <head>...</head>\n  <body>\n   <body\">\n       <a name=\"top\"></a>\n       <h1>VCWPD Area Reservoir Report</h1>\n       <table border=\"1\" id=\"weather_table\">\n         <tbody>\n           <tr class=\"header\">...</tr>\n           <tr class=\"row0\">...</tr>\n           <tr class=\"row1\">\n             <td>691</td>\n             <td>Lake Casitas Dam level</td>\n             <td>11/29/2019 09:29</td>\n             <td align=\"Right\">567.00</td>\n             <td align=\"Right\">238,000</td>\n             <td align=\"Right\">500.34</td>\n             <td align=\"Right\">98,037</td>\n             <td align=\"Right\">---</td>\n             <td align=\"Right\">0</td>\n             <td align=\"Right\">-66.66</td>\n             <td align=\"Right\">41.2%</td>\n            </tr>\n           <tr class=\"row0\">...</tr>\n           <tr class=\"row1\">...</tr>\n           <tr class=\"row0\">...</tr>\n           <tr class=\"row1\">...</tr>\n     </tbody>\n    </table>\n   </body\">\n  </body>\n</html>","output":"str","x":260,"y":720,"wires":[["f5934049.fb9f7","77148927.a91428"]]},{"id":"f5934049.fb9f7","type":"html","z":"ea593186.4918d","name":"","property":"payload","outproperty":"payload","tag":"#weather_table > tbody > tr:nth-child(3)","ret":"html","as":"single","x":320,"y":820,"wires":[["a67f38ab.0cdd98"]]},{"id":"77148927.a91428","type":"debug","z":"ea593186.4918d","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":550,"y":700,"wires":[]},{"id":"a67f38ab.0cdd98","type":"debug","z":"ea593186.4918d","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":550,"y":760,"wires":[]}]
3

Thank you for the reply @dceejay that does work. But when I link it to the actual site it comes up null. refer to edit of original question for copy of flow

4

Thanks @dceejay for you help, I believe it is because I need to add a slight delay to the http request so to allow for the html table to populate before parsing data.
Do I need to do this via a function node or can I use OOTB nodes?

5

I figured it out, I recreated the html code in a template node and extended the calls to become http/s addresses including the json address for data.

image
image900Ă—175 9.04 KB

From here I can drill down into the data to the locations I am after and manipulate as I need.
Flow below:

[{"id":"f1113a48.b605d8","type":"inject","z":"df68c0a8.8752","name":"get data","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"x":420,"y":500,"wires":[["55ba596a.dfa538"]]},{"id":"11b1bcbf.c044d3","type":"debug","z":"df68c0a8.8752","name":"","active":true,"complete":false,"x":930,"y":500,"wires":[]},{"id":"55ba596a.dfa538","type":"ui_template","z":"df68c0a8.8752","group":"e2ee0272.3124","name":"Reservoir Table","order":13,"width":0,"height":0,"format":"<!DOCTYPE html><HTML>\n<HEAD>\n<Title>VCWPD - Area Reservoir Report</TITLE>\n\n<!-- Style Sheets -->\n<link rel=\"stylesheet\" href=\"https://www.vcwatershed.net/fws/css/reservoir.css\">\n\n\n<!-- Jquery and Jquery UI -->\n<script src=\"https://www.vcwatershed.net/js/libs/jquery-1.7.2.min.js\"></script>\n\n<!-- Javascript Libraries -->\n\n<!-- D3.js Libraries -->\n<script type=\"text/javascript\" src=\"https://www.vcwatershed.net/fws/js/d3.v3.min.js\"></script>\n\n<script language=\"JavaScript\">\nvar resData = null;\n\nfunction addCommas(nStr)\n{\n\tnStr += '';\n\tx = nStr.split('.');\n\tx1 = x[0];\n\tx2 = x.length > 1 ? '.' + x[1] : '';\n\tvar rgx = /(\\d+)(\\d{3})/;\n\twhile (rgx.test(x1)) {\n\t\tx1 = x1.replace(rgx, '$1' + ',' + '$2');\n\t}\n\treturn x1 + x2;\n}\n\nfunction GetDamInfo(resInfo, id) {\n\tvar i;\n\tvar data = null;\n\t\n\tfor (i=0; i<resInfo.length; i++){\n\t\tif (id == resInfo[i].Site) {\n\t\t\tdata = resInfo[i];\n\t\t}\n\t}\n\treturn data;\n}\n\n$(document).ready(function() {\n\tvar row = 0;\n\tvar i = 0;\n\tvar dateTime = '';\n\tvar classId = '';\n\tvar stage = '';\n\tvar curVol = \"\";\n\tvar outflow = \"\";\n\tvar spillFlow = \"\";\n\tvar space = \"\";\n\tvar storage = '';\n\tvar spillElev = '';\n\tvar perFull = '';\n\tvar belowSpill = '';\n\t\n\tvar resInfo = [{\"Site\":\"690\",\"Spillway\":1095.38,\"Volume\":1503},{\"Site\":\"691\",\"Spillway\":567.00,\"Volume\":238000},{\"Site\":\"789\",\"Spillway\":2579.00,\"Volume\":171196},{\"Site\":\"792\",\"Spillway\":1055.00,\"Volume\":81986},{\"Site\":\"990\",\"Spillway\":1515.00,\"Volume\":323696},{\"Site\":\"890\",\"Spillway\":1227.00,\"Volume\":1235},{\"Site\":\"891\",\"Spillway\":797.00,\"Volume\":670}];\n\t\n\t$.when(\n\t\t$.getJSON('https://www.vcwatershed.net/fws/json/reservoir.json', function(data) {resData=data;})\t\n\t).then(function(){\n\t\tif (resData) {\n\t\t\t$.each(resData.data, function() {\n\t\t\t\tif (this.site != 'none') {\n\t\t\t\t\trow = (i % 2);\n\t\t\t\t\tclassId = \"row\" + row;\n\t\t\t\t\tif (this.last_rpt == '-999T-999') {\n\t\t\t\t\t\tdateTime = '---';\n\t\t\t\t\t} else {\n\t\t\t\t\t\tdateTime = this.last_rpt.substr(5,2) + '/' + this.last_rpt.substr(8,2) + '/' + this.last_rpt.substr(0,4) + ' ' + this.last_rpt.substr(11,2) + ':' + this.last_rpt.substr(14,2);\n\t\t\t\t\t}\n\t\t\t\t\tif (this.stage == -999) {\n\t\t\t\t\t\tstage = '---';\n\t\t\t\t\t} else {\n\t\t\t\t\t\tstage = this.stage;\n\t\t\t\t\t}\n\t\t\t\t\tif (this.volume == -999) {\n\t\t\t\t\t\tcurVol = '---';\n\t\t\t\t\t} else {\n\t\t\t\t\t\tcurVol = this.volume;\n\t\t\t\t\t}\n\t\t\t\t\tif (this.outflow == -999) {\n\t\t\t\t\t\toutflow = '---';\n\t\t\t\t\t} else {\n\t\t\t\t\t\toutflow = this.outflow;\n\t\t\t\t\t}\n\t\t\t\t\tif (this.spillway == -999) {\n\t\t\t\t\t\tspillFlow = '---';\n\t\t\t\t\t} else {\n\t\t\t\t\t\tspillFlow = this.spillway;\n\t\t\t\t\t}\n\t\t\t\t\tvar info = GetDamInfo(resInfo,this.site);\n\t\t\t\t\tif (info == null) {\n\t\t\t\t\t\tstorage = '---';\n\t\t\t\t\t\tspillElev = '---';\n\t\t\t\t\t\tperFull = '---';\n\t\t\t\t\t} else {\n\t\t\t\t\t\tstorage = info.Volume;\n\t\t\t\t\t\tspillElev = info.Spillway;\n\t\t\t\t\t\tif (curVol != '---') {\n\t\t\t\t\t\t\tperFull = (curVol/storage) * 100;\n\t\t\t\t\t\t\tperFull = perFull.toFixed(1) + '%';\n\t\t\t\t\t\t} else {\n\t\t\t\t\t\t\tperFull = '---';\t\n\t\t\t\t\t\t}\n\t\t\t\t\t\tif (stage != '---') {\n\t\t\t\t\t\t\tbelowSpill = stage - spillElev;\n\t\t\t\t\t\t\tbelowSpill = addCommas(belowSpill.toFixed(2));\n\t\t\t\t\t\t} else {\n\t\t\t\t\t\t\tbelowSpill = '---';\n\t\t\t\t\t\t}\n\t\t\t\t\t\tspillElev = addCommas(spillElev.toFixed(2));\n\t\t\t\t\t\tstorage = addCommas(storage);\n\t\t\t\t\t}\n\t\t\t\t\tif (curVol != '---') {\n\t\t\t\t\t\tcurVol = addCommas(curVol);\n\t\t\t\t\t}\n\t\t\t\t\tif (stage != '---') {\n\t\t\t\t\t\tstage = addCommas(stage.toFixed(2));\n\t\t\t\t\t}\n\t\t\t\t\t$('#reservoir_table > tbody').append('<TR class=\"' + classId + '\"><TD>' + this.site + '</TD><TD>' + this.name + '</TD><TD>' + dateTime + '</TD><TD align=\"Right\">' + spillElev + '</TD><TD align=\"Right\">' + storage + '</TD><TD align=\"Right\">' + stage + '</TD><TD align=\"Right\">' + curVol + '</TD><TD align=\"Right\">' + outflow + '</TD><TD align=\"Right\">' + spillFlow + '</TD><TD align=\"Right\">' + belowSpill + '</TD><TD align=\"Right\">' + perFull + '</TD></TR>');\n\t\t\t\t\ti++;\n\t\t\t\t}\n\t\t\t});\n\t\t}\n\t\n\t});\n\n});\n\n</script>\n\n</HEAD>\n<body\">\n<a name=\"top\"></a>\n<H1>VCWPD Area Reservoir Report</H1>\n\n<Table  border=1  id=\"reservoir_table\">\n<tbody>\n<TR class=\"header\"><TH>Site</TH><TH>Name</TH><TH>Date/Time (PST)</TH><TH>Spillway Elev (ft)</TH><TH>Capacity (AF)</TH><TH>Current Stage (ft)</TH><TH>Current Volume (AF)</TH><TH>Outflow (cfs)</TH><TH>Spill Flow (cfs)</TH><TH>Below Spill (ft)</TH><TH>% Capacity</TH></TR>\n</tbody>\n</Table>\n</Body>\n</HTML>\n","storeOutMessages":true,"fwdInMessages":true,"templateScope":"local","x":680,"y":500,"wires":[["11b1bcbf.c044d3"]]},{"id":"e2ee0272.3124","type":"ui_group","z":"","name":"News","tab":"aaa98c79.de1be","order":3,"disp":true,"width":20,"collapse":false},{"id":"aaa98c79.de1be","type":"ui_tab","z":"","name":"Home","icon":"home","order":1,"disabled":false,"hidden":false}]```
1 Like
6

Nicely done !