# HTML is NOT parsing DIV tags consistently

**URL:** <https://discourse.nodered.org/t/html-is-not-parsing-div-tags-consistently/29417>\
**Category:** General\
**Created:** [3 July 2020 03:18 UTC](https://discourse.nodered.org/t/html-is-not-parsing-div-tags-consistently/29417 "2020-07-03T03:18:07Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![nedstrader](https://avatars.discourse-cdn.com/v4/letter/n/4491bb/32.png) [@nedstrader](https://discourse.nodered.org/u/nedstrader)\
**Post date:** [3 July 2020 03:18 UTC](https://discourse.nodered.org/t/html-is-not-parsing-div-tags-consistently/29417/1 "2020-07-03T03:18:07Z")

</div>

Hello,

I have a flow that scrapes the U.S. Securities website page: "sec.report"

This web page contains a Company List table that contains the following columns:

1. Company Name. ( Como Health Holdings, LLC )
2. State Abbreviation Code. ( MO = Montana )
3. State of Incorporation Abbreviation Code. ( MO = Montana )
4. CIK - Central Index Key ( CIK = 0001816732 )
5. Updated. ( Updated = 2020-07-02 )

NODE-RED FLOW:

1. HTTP Request Node

- Gets the web page data and returns a string.
- URL = [https://sec.report](https://sec.report)

1. HTML Node with the following selector:  
Selector = div.table\>div\>div\>div

2. When Node-Red processed the web page, it did NOT generate a carriage  
return for the two state columns 2 and 3.

I have attached two screen prints:

1. SEC Report - Node Red.png
2. SEC Report - web page and inspect window.png

Does anyone know why the carriage returns are not being generated by the div tags and how to fix the string elements in the string array ???

Ideally, I would like to convert the string elements into separate key=value pairs and create an object to replace the string element in the array.

Thanks  
Vic

 ![SEC Report - Node Red](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/2/6/2635bfcb134e49891191e4ab6d57db79bb4d59cd.png) ![SEC Report - web page and inspect window](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/a/9/a9c2346eb55e1d2de5ca2adf5d544179a6b8660f.png)

---

<div class="post-metadata">

**Author:** ![bakman2](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bakman2/32/6207_2.png) [@bakman2](https://discourse.nodered.org/u/bakman2)\
**Post date:** [3 July 2020 05:11 UTC](https://discourse.nodered.org/t/html-is-not-parsing-div-tags-consistently/29417/2 "2020-07-03T05:11:57Z")

</div>

There is no need for the json node.

Try this flow (bit rudimentary way of doing it, but it works):

```auto
[{"id":"6f07b3fa.94448c","type":"inject","z":"5de8b0ff.e7f7a","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":180,"y":384,"wires":[["94bfc134.ad2ce"]]},{"id":"94bfc134.ad2ce","type":"http request","z":"5de8b0ff.e7f7a","name":"","method":"GET","ret":"txt","paytoqs":false,"url":"https://sec.report","tls":"","persist":false,"proxy":"","authType":"","x":334,"y":384,"wires":[["5f1c3b20.c1cf2c"]]},{"id":"5f1c3b20.c1cf2c","type":"html","z":"5de8b0ff.e7f7a","name":"","property":"payload","outproperty":"payload","tag":"div.table .row .cell","ret":"text","as":"single","x":522,"y":384,"wires":[["f15144b9.0fd3a"]]},{"id":"87b90c36.4b8a78","type":"debug","z":"5de8b0ff.e7f7a","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":862,"y":384,"wires":[]},{"id":"f15144b9.0fd3a","type":"function","z":"5de8b0ff.e7f7a","name":"","func":"m = msg.payload\no = []\nfor(x=0;x<m.length;x+=5){\n \n \n company = m[x].trim()\n state =m[x+1].trim()\n inc = m[x+2].trim()\n cik = m[x+3].trim()\n updated = m[x+4].trim()\n \n o.push({'company':company,'state':state,'inc':inc,'cik':cik,'updated':updated})\n \n \n}\no.shift()\nreturn {payload:o}","outputs":1,"noerr":0,"x":698,"y":384,"wires":[["87b90c36.4b8a78"]]}]

```

output

```auto
[
  {
    "company": "Culinary Craft Workshop, LLC",
    "state": "MD",
    "inc": "MD",
    "cik": "0001816758",
    "updated": "2020-07-02"
  },
  {
    "company": "Thomas Robert A",
    "state": "",
    "inc": "",
    "cik": "0001816757",
    "updated": "2020-07-02"
  },
  {
    "company": "Green Jeffrey D",
    "state": "",
    "inc": "",
    "cik": "0001816753",
    "updated": "2020-07-02"
  },
...

```

---

<div class="post-metadata">

**Author:** ![nedstrader](https://avatars.discourse-cdn.com/v4/letter/n/4491bb/32.png) [@nedstrader](https://discourse.nodered.org/u/nedstrader)\
**Post date:** [3 July 2020 12:45 UTC](https://discourse.nodered.org/t/html-is-not-parsing-div-tags-consistently/29417/3 "2020-07-03T12:45:49Z")

</div>

@bakman2

🕶 🥳

This is Fantastic !

I am new to Node-Red and I have been trying to understand how to interact with the node flow and this is the best explanation that I have seen.

Just Bypass the node processing with a function node.

I guess there are no For Loop Processing Nodes where you can specify the criteria and the for loop body statements ???

Thanks  
nedstrader

---

<div class="post-metadata">

**Author:** ![nedstrader](https://avatars.discourse-cdn.com/v4/letter/n/4491bb/32.png) [@nedstrader](https://discourse.nodered.org/u/nedstrader)\
**Post date:** [3 July 2020 13:23 UTC](https://discourse.nodered.org/t/html-is-not-parsing-div-tags-consistently/29417/4 "2020-07-03T13:23:41Z")

</div>

@bakman2 would you say that there is something wrong with the HTML node where it is skipping those two div fields ?

Should we create a ticket ?

---

<div class="post-metadata">

**Author:** ![bakman2](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bakman2/32/6207_2.png) [@bakman2](https://discourse.nodered.org/u/bakman2)\
**Post date:** [3 July 2020 15:39 UTC](https://discourse.nodered.org/t/html-is-not-parsing-div-tags-consistently/29417/5 "2020-07-03T15:39:33Z")

</div>

> Should we create a ticket ?

No.

empty divs returned linebreak.

> I guess there are no For Loop Processing Nodes where you can specify the criteria and the for loop body statements ???

Could have used a split node as well, but because the data returns line-breaks this is not nice to deal with the split node.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [17 July 2020 15:39 UTC](https://discourse.nodered.org/t/html-is-not-parsing-div-tags-consistently/29417/6 "2020-07-17T15:39:37Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.