\
**Category:** Developing Nodes\
**Created:** [4 January 2024 17:28 UTC](https://discourse.nodered.org/t/question-about-html-node/84298 "2024-01-04T17:28:56Z")\
**Posts on this page:** 20\
**Page:** 1
**Author:**  [@gregorius](https://discourse.nodered.org/u/gregorius)\
**Post date:** [4 January 2024 17:28 UTC](https://discourse.nodered.org/t/question-about-html-node/84298/1 "2024-01-04T17:28:56Z")
Hi There,
I'm just utilising the html node and have a question about the options:
with those three options, it's not possible to get the attributes _and_ the content.
For example with the selector being 'a':
```auto
some text
```
- with the first option "html content", I get `some text` as payload
- second option, payload becomes `some text`, and
- third option: `{ href: 'url', 'target': '_blank' }`
What I would like to have is something like:
```json
{ href: 'url',
'target': '_blank',
_: "some text"
}
```
is that possible? What am I doing wrong?
Thanks for any tips ...
---
**Author:**  [@UnborN](https://discourse.nodered.org/u/UnborN)\
**Post date:** [4 January 2024 18:00 UTC](https://discourse.nodered.org/t/question-about-html-node/84298/2 "2024-01-04T18:00:11Z")
What if you use the Html node twice ?
Once to get the `attributes` and one for the `content` and
then use a Join node to merge the results in one msg
**Test Flow :**
```auto
[{"id":"3bb37f38619ad9e9","type":"inject","z":"54efb553244c241f","name":"html","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"some text","payloadType":"str","x":270,"y":3180,"wires":[["b332d974fcb36acd","71c95918952af649"]]},{"id":"b332d974fcb36acd","type":"html","z":"54efb553244c241f","name":"attributes","property":"payload","outproperty":"payload","tag":"a","ret":"attr","as":"multi","x":460,"y":3140,"wires":[["5d8eb3e4bf5506ae"]]},{"id":"6cd4a1f22a03b652","type":"debug","z":"54efb553244c241f","name":"debug 59","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":860,"y":3180,"wires":[]},{"id":"71c95918952af649","type":"html","z":"54efb553244c241f","name":"content","property":"payload","outproperty":"payload","tag":"a","ret":"html","as":"multi","x":460,"y":3220,"wires":[["3a885d7c3413c462"]]},{"id":"b1508985cc7d5986","type":"join","z":"54efb553244c241f","name":"","mode":"custom","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"2","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":690,"y":3180,"wires":[["6cd4a1f22a03b652"]]},{"id":"5d8eb3e4bf5506ae","type":"change","z":"54efb553244c241f","name":"","rules":[{"t":"set","p":"topic","pt":"msg","to":"attributes","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":555,"y":3140,"wires":[["b1508985cc7d5986"]],"l":false},{"id":"3a885d7c3413c462","type":"change","z":"54efb553244c241f","name":"","rules":[{"t":"set","p":"topic","pt":"msg","to":"content","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":555,"y":3220,"wires":[["b1508985cc7d5986"]],"l":false}]
```
**Result :**
---
**Author:**  [@gregorius](https://discourse.nodered.org/u/gregorius)\
**Post date:** [4 January 2024 18:16 UTC](https://discourse.nodered.org/t/question-about-html-node/84298/3 "2024-01-04T18:16:05Z")
> [@UnborN](#):
>
> What if you use the Html node twice ?
Sure I was thinking that too .... but that would assume that the underlying Cheerio lib never changes and will _always_ provide the same order ... .oO(what could possibly go wrong?!?) 😉
plus it would mean parsing the same content twice and that isn't exactly efficient.
I was looking at the [implementation](https://github.com/node-red/node-red/blob/0e8d3127944a91f39fb49bcb4b388b184ce11edd/packages/node_modules/%40node-red/nodes/core/parsers/70-HTML.js#L44-L50) and before I parse the content twice and merge and hope the order is the same (since there isn't anything else to link the data), I would just roll my own! 🌡
Thanks for the idea/approach 👍
---
**Author:**  [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [4 January 2024 18:18 UTC](https://discourse.nodered.org/t/question-about-html-node/84298/4 "2024-01-04T18:18:36Z")
Happy for anyone wanting to enhance the node with a PR 🙂
---
**Author:**  [@gregorius](https://discourse.nodered.org/u/gregorius)\
**Post date:** [4 January 2024 18:29 UTC](https://discourse.nodered.org/t/question-about-html-node/84298/5 "2024-01-04T18:29:54Z")
> [@dceejay](#):
>
> anyone wanting to enhance
Hm. Then give me an idea how the core team would like to do this?
Looking at the [code](https://github.com/node-red/node-red/blob/0e8d3127944a91f39fb49bcb4b388b184ce11edd/packages/node_modules/%40node-red/nodes/core/parsers/70-HTML.js#L44-L50):
```auto
if (node.ret === "html") { pay2 = cheerio.load($(this).html().trim(),null,false).xml(); }
if (node.ret === "text") { pay2 = $(this).text(); }
if (node.ret === "attr") {
pay2 = Object.assign({},this.attribs);
}
//if (node.ret === "val") { pay2 = $(this).val(); }
```
Is `val()` actually the complete element? I don't know what Cheerio `val()` return ...
Would something like:
```js
if ( node.ret === "complete" ) {
pay2 = Object.assign({ _: $(this).html() },this.attribs);
}
```
would do the job or is `_` reserved? From memory, xml parsers usually use `_` as reference to the contents:
Btw: I could also use the XML to parse the original HTML but the html is broken:
---
**Author:**  [@gregorius](https://discourse.nodered.org/u/gregorius)\
**Post date:** [4 January 2024 19:31 UTC](https://discourse.nodered.org/t/question-about-html-node/84298/6 "2024-01-04T19:31:26Z")
One more question, is this [line](https://github.com/node-red/node-red/blob/0e8d3127944a91f39fb49bcb4b388b184ce11edd/packages/node_modules/%40node-red/nodes/core/parsers/70-HTML.js#L33)
```auto
if (msg.hasOwnProperty("select")) { tag = node.tag || msg.select; }
```
wrong and shouldn't it be
```auto
if (msg.hasOwnProperty("select")) { tag = msg.select || node.tag; }
```
since else if the tag is sent on the node _and_ the msg, the the node will continue to be used? I thought the msg overrides the values of nodes?
---
**Author:**  [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [4 January 2024 19:32 UTC](https://discourse.nodered.org/t/question-about-html-node/84298/7 "2024-01-04T19:32:16Z")
> [@gregorius](#):
>
> Is `val()` actually the complete element? I don't know what Cheerio `val()` return ...
No, Cheerio follows jQuery-like syntax but `val()` only returns the value of an input, select or textarea.
To get slot content (effectively the `innerHTML`) and attributes at the same time seems to require 2 calls, one to `attr()` and one to `html()` (or `text()`).
---
**Author:**  [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [4 January 2024 19:52 UTC](https://discourse.nodered.org/t/question-about-html-node/84298/8 "2024-01-04T19:52:19Z")
> [@gregorius](#):
>
> would do the job
... worth a try 🙂 and the \_ should be as per how the user configures it - so yes \_ by default.
---
**Author:**  [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [4 January 2024 19:54 UTC](https://discourse.nodered.org/t/question-about-html-node/84298/9 "2024-01-04T19:54:29Z")
> [@gregorius](#):
>
> is this .... wrong
hmm no - generally - by default if the user has set a parameter in the config then the msg should not override it. If they leave it blank (or unset) then yes it can be overridden. So the order is correct as-is
---
**Author:**  [@gregorius](https://discourse.nodered.org/u/gregorius)\
**Post date:** [4 January 2024 21:12 UTC](https://discourse.nodered.org/t/question-about-html-node/84298/10 "2024-01-04T21:12:12Z")
So I created a node to replace the original node - originally I called it [html2](https://github.com/gorenje/node-red-contrib-html-proto/tree/main) but only for purposes of testing my changes.
It does what it's meant to do but before I submit a PR, I just want to check that code is ok:
- including the html to attrib hash: [here](https://github.com/gorenje/node-red-contrib-html-proto/blob/ec356be5bb52d8db9f44d82f1ab9bde363540033/nodes/html2.js#L53-L57) and [here](https://github.com/gorenje/node-red-contrib-html-proto/blob/ec356be5bb52d8db9f44d82f1ab9bde363540033/nodes/html2.js#L80-L84)
- adding the `chr` (name taken from the xml node) property for [defining the attr name](https://github.com/gorenje/node-red-contrib-html-proto/blob/ec356be5bb52d8db9f44d82f1ab9bde363540033/nodes/html2.js#L28)
- added the [option](https://github.com/gorenje/node-red-contrib-html-proto/blob/ec356be5bb52d8db9f44d82f1ab9bde363540033/nodes/html2.html#L22)
- added `chr` field [here](https://github.com/gorenje/node-red-contrib-html-proto/blob/ec356be5bb52d8db9f44d82f1ab9bde363540033/nodes/html2.html#L57) and [here](https://github.com/gorenje/node-red-contrib-html-proto/blob/ec356be5bb52d8db9f44d82f1ab9bde363540033/nodes/html2.html#L40-L43)
- and moved [name field to top](https://github.com/gorenje/node-red-contrib-html-proto/blob/ec356be5bb52d8db9f44d82f1ab9bde363540033/nodes/html2.html#L2-L6)
The UI changed by moving the name field to the top but also there is a fourth option:
and the field for the attribute is shown if the fourth option is selected:
but hidden when not:
---
**Author:**  [@UnborN](https://discourse.nodered.org/u/UnborN)\
**Post date:** [4 January 2024 22:13 UTC](https://discourse.nodered.org/t/question-about-html-node/84298/11 "2024-01-04T22:13:30Z")
Nice work
Since you are in the code, have the experience and planning a PR 😉
can you check why when we set a new field for the Output it doesnt seem to work correctly ?
For example i want the output in
doesnt produce the correct result .. it just outputs the original `msg.payload`
Test html2 flow
```auto
[{"id":"3bb37f38619ad9e9","type":"inject","z":"54efb553244c241f","name":"html","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"true","payloadType":"bool","x":310,"y":3380,"wires":[["80f45e51097c7ded"]]},{"id":"db6cb891535b2186","type":"debug","z":"54efb553244c241f","name":"debug 60","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":780,"y":3380,"wires":[]},{"id":"30216608caf646c7","type":"html2","z":"54efb553244c241f","name":"","property":"payload","outproperty":"payload.newField","tag":"a","ret":"compl","as":"single","chr":"_","x":610,"y":3380,"wires":[["db6cb891535b2186"]]},{"id":"80f45e51097c7ded","type":"template","z":"54efb553244c241f","name":"","field":"payload","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"aaaaaa\nbbbbbb","output":"str","x":460,"y":3380,"wires":[["30216608caf646c7"]]}]
```
[EDIT]
must be because the original payload is a string and not an object and we cannot tag on another new key field to it ?
---
**Author:**  [@gregorius](https://discourse.nodered.org/u/gregorius)\
**Post date:** [4 January 2024 22:52 UTC](https://discourse.nodered.org/t/question-about-html-node/84298/12 "2024-01-04T22:52:07Z")
> [@UnborN](#):
>
> original payload is a string
correct, try using something like `msg.snafu.fubar` as output property and you get what you want.
The thing doing the setting is this [line](https://github.com/node-red/node-red/blob/0e8d3127944a91f39fb49bcb4b388b184ce11edd/packages/node_modules/%40node-red/nodes/core/parsers/70-HTML.js#L54) and `setMessageProperty` handles about any magic you throw at it! But not if the attribute exists and you expect it to be an object when its a string.
So I would say it's not a bug but a feature! 😉
---
**Author:**  [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [4 January 2024 23:54 UTC](https://discourse.nodered.org/t/question-about-html-node/84298/13 "2024-01-04T23:54:48Z")
> [@UnborN](#):
>
> doesnt produce the correct result .. it just outputs the original `msg.payload`
You have to change msg.payload into an object before you can add a property to it. As @gregorius says, it is because msg.payload already exists as a string and you cannot add a property to a string.
---
**Author:**  [@bakman2](https://discourse.nodered.org/u/bakman2)\
**Post date:** [5 January 2024 04:36 UTC](https://discourse.nodered.org/t/question-about-html-node/84298/14 "2024-01-05T04:36:27Z")
This is great stuff.
In the "previous" html node (years ago), there was an xpath option as well, which was very useful, because when capturing from the web inspector, one can copy the xpath which could immediately be used/replicated within NR, while the selector copy only gets the class/id and does not traverse into the DOM tree.
---
**Author:**  [@gregorius](https://discourse.nodered.org/u/gregorius)\
**Post date:** [5 January 2024 08:48 UTC](https://discourse.nodered.org/t/question-about-html-node/84298/15 "2024-01-05T08:48:02Z")
> [@bakman2](#):
>
> there was an xpath option as well,
I could imagine - but I am only guessing - that disappeared because you can create a function node with cheerio and then do your own parsing of html content:
```auto
[{"id":"e44d1e4a55be1eee","type":"function","z":"07755b701d94ac63","name":"function 32","func":"var $ = cheerio.load(msg.payload)\n\n$(\"a[title=Sitemap]\").each( (idx,e) => {\n node.send({\n ...msg,\n payload: Object.assign({ _: $(e).html().trim() }, e.attribs)\n })\n})\n","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[{"var":"cheerio","module":"cheerio"}],"x":1254,"y":3237,"wires":[["9f4deae00a9b478f"]]}]
```
this function node searches for a "Sitemap" link and emits it when found. I believe function nodes couldn't include third party modules for a long time? (It could also be that the html node changed the underlying library from xpath to cheerio - cheerio doesn't really support xpath)
Of course this is a perfect replacement for what I did above and it is also a reminder that most nodes within Node-RED can/could be replaced by a well written function nodes. Hence I would - to a certain extent - argue that the above changes made to the html node do not make it into the core since my opinion is that the core should be as thin/stable/solid as possible with extra functionality being added via plugins.
Why? Because the core should be as stable as possible without random failures and constant updating. Plugins are simpler to debug, fix and update then having to patch the core, followed by making a new release and finally have everyone update their installations.
I don't have to update my Linux just because Firefox has a bug - that's how I see Node-RED and its plugins. Linux has the [C standard library](https://en.wikipedia.org/wiki/C_standard_library) so it's clear what Linux has to provide as APIs - this isn't so clear for Node-RED since there is no "base" functionality that is standardised. So should the html node offer a fourth option or not? No standard defines a html node and its functionality.
Sorry for the rant, snowy cold morning here ☹
---
**Author:**  [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [5 January 2024 14:23 UTC](https://discourse.nodered.org/t/question-about-html-node/84298/16 "2024-01-05T14:23:05Z")
> [@gregorius](#):
>
> Sorry for the rant, snowy cold morning here
No snow here, just cold. Though myself and Mrs K. have been planning an Arctic Circle trip today - we hope to be visiting Finland or Norway soon. 🥶
* * *
But more on-topic, while I mostly agree with what you've said, I think there would be room for an option that recovers both an element's content and its attributes. I don't think that is unreasonable and would be complex for a non-coder to achieve. But yes, grabbing content from an HTML page can get arbitrarily complex, needing complex queries which are best served by code.
---
**Author:**  [@gregorius](https://discourse.nodered.org/u/gregorius)\
**Post date:** [5 January 2024 15:02 UTC](https://discourse.nodered.org/t/question-about-html-node/84298/17 "2024-01-05T15:02:07Z")
> [@TotallyInformation](#):
>
> to be visiting Finland or Norway soon
I don't need to, they've come to me!
> [@TotallyInformation](#):
>
> both an element's content and its attributes.
Ok, so you would be for a fourth option on the existing html node but no xpath specification? I would see that as a good option ...
btw just hit the utf-8 v. iso-8859-1 encoding on CSV data problem ... should the CSV node handle that? No, for there is the [iconv node](https://flows.nodered.org/node/node-red-contrib-iconv)!
---
**Author:**  [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [5 January 2024 16:07 UTC](https://discourse.nodered.org/t/question-about-html-node/84298/18 "2024-01-05T16:07:22Z")
> [@gregorius](#):
>
> Ok, so you would be for a fourth option on the existing html node but no xpath specification? I would see that as a good option ...
Honestly, I'm not fussed about an xpath option. Not against it, just not interested in it. 🙂
I've done more than my fair share of xpath and I've no intention of ever going back to it!! The world moved on and so did the HTML spec. CSS Selectors are the way forward and that's all I use.
---
**Author:**  [@gregorius](https://discourse.nodered.org/u/gregorius)\
**Post date:** [5 January 2024 16:15 UTC](https://discourse.nodered.org/t/question-about-html-node/84298/19 "2024-01-05T16:15:11Z")
> [@TotallyInformation](#):
>
> CSS Selectors are the way forward
xpaths are still a big part of scraping data from websites - I'm just playing around with data-crawler implementation in Node-RED and xpaths are definitely useful. Perhaps that is also related to this being server-side and not in the browser.
Cheerio is a big exception since it implements the jQuery API on the server side - mind-blowing to be honest!
---
**Author:**  [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [5 January 2024 16:26 UTC](https://discourse.nodered.org/t/question-about-html-node/84298/20 "2024-01-05T16:26:03Z")
As I say, I'm not against xpath - just had too much of it in the bad-old days when XML ruled the world - before JSON was a think.
> [@gregorius](#):
>
> Cheerio is a big exception since it implements the jQuery API on the server side - mind-blowing to be honest!
Yes, I even considered it for the `uib-html` node but `jsdom` was the better and more comprehensive choice for being able to convert from a JSON schema to HTML.
[Next page](https://discourse.nodered.org/t/question-about-html-node/84298.md?page=2)