Help with getting information from a site - "scraping"?

Trying_to_learn · 12 January 2024 05:11

Hi folks.

The John Dory:
I live in Oz and have foxtel.

I also have a flow in NR that automatically changes channels when my favorite show is on.
(Recording isn't a problem as it is all done on their box)

Anyway.....

The EXACT time varies - sometimes, and it is annoying either missing the start (not a big deal, as I can start over) but anyway.....

I was thinking that I set a NOMINAL time (enough before when it will start to not miss the start) and at that time the EPG is parsed and the PUBLISHED time is taken and the time the channel changes is based on that.

I may take off 2 minutes just to be sure. But at this time, that is a long way down the track.

Help - please.
How would I do it.

And before you ask:
No, I've never tried and don't have the URL for the timetable/program yet.
Not much good getting it if I can't use it.

GogoVega · 12 January 2024 08:26

Hi,
If you summarize it, what does it give?

When I was recording (was a time) I added 15 minutes before and after what the timetable said

smcgann99 · 12 January 2024 12:52

The time offset is not difficult, but exactly how to get the info does depend on the source, so you need to show that link before anyone can be more specific about how you might scrape the data.

Generally you would use the http request node to get the data and the html node to parse it.

Trying_to_learn · 12 January 2024 19:40

You seem to have not understood the question.

I am NOT wanting to record.

I want to get the time table and parse it for the start time, so a couple of minutes before that time the TV changes to that channel.

Trying_to_learn · 12 January 2024 19:40

Yeah, ok.

I guess it is a kind of catch 22 scenario

Trying_to_learn · 12 January 2024 20:36

This is a link which shows me what I need.

https://www.foxtel.com.au/now/tv-guide/grid

marcus-j-davies · 12 January 2024 21:24

I might have a way..... Testing....

marcus-j-davies · 12 January 2024 21:36

I think I have it

Let's say I want to get the time of Blue Bloods? the selector will be

$("a[href^=Blue-Bloods] > p.sr-only").first().text()

The above will return "8:50am"

Screenshot 2024-01-12 at 21.36.24

I think you can use the HTML node for it

EDIT
It does look like this website doesn't play nice with the Request node - so your milage may very

UnborN · 12 January 2024 22:16

Im not sure about the accuracy of the html selectors but you can play around with this :

[{"id":"305cb71031a34412","type":"inject","z":"54efb553244c241f","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":140,"y":4180,"wires":[["80a7a0c01b20e948"]]},{"id":"80a7a0c01b20e948","type":"http request","z":"54efb553244c241f","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://www.foxtel.com.au/now/tv-guide/grid","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[{"keyType":"Accept","keyValue":"","valueType":"text/html","valueValue":""},{"keyType":"User-Agent","keyValue":"","valueType":"other","valueValue":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0"}],"x":310,"y":4180,"wires":[["08950add9b13deb4","d80d68b711c5ebfd"]]},{"id":"08950add9b13deb4","type":"html","z":"54efb553244c241f","name":"","property":"payload","outproperty":"payload","tag":".epg-cell .epg-program-name","ret":"text","as":"single","x":590,"y":4120,"wires":[["bfd1390fc9a8d0ba"]]},{"id":"dd139d12ce4a965e","type":"debug","z":"54efb553244c241f","name":"debug 62","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1220,"y":4160,"wires":[]},{"id":"d80d68b711c5ebfd","type":"html","z":"54efb553244c241f","name":"","property":"payload","outproperty":"payload","tag":" .epg-cell .sr-only","ret":"html","as":"single","x":550,"y":4200,"wires":[["474cf6d19ead5ed7"]]},{"id":"c7013900160e3c9e","type":"join","z":"54efb553244c241f","name":"","mode":"custom","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"2","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":890,"y":4160,"wires":[["d69cf8bac35572c2"]]},{"id":"bfd1390fc9a8d0ba","type":"change","z":"54efb553244c241f","name":"","rules":[{"t":"set","p":"topic","pt":"msg","to":"title","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":755,"y":4120,"wires":[["c7013900160e3c9e"]],"l":false},{"id":"474cf6d19ead5ed7","type":"change","z":"54efb553244c241f","name":"","rules":[{"t":"set","p":"topic","pt":"msg","to":"time","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":755,"y":4200,"wires":[["c7013900160e3c9e"]],"l":false},{"id":"d69cf8bac35572c2","type":"function","z":"54efb553244c241f","name":"function 3","func":"msg.payload = msg.payload.title.map( (m,i) => {\n    return { \n        movie: m.trim(),\n        time: msg.payload.time[i]\n    }\n})\n\nreturn msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":1040,"y":4160,"wires":[["dd139d12ce4a965e"]]}]

It does two separate filtrations .. one for the title and one for the time and then merges them into one array

.. that you can then search, for a specific "title" to get a matching start time.

Trying_to_learn · 13 January 2024 20:35

Looking good.

But I shall admit I am still a bit confused.

I'm guessing this downloads the entire EPG for all channels.
Slowly going through the output.

But I'm not sure how I can get the channel name also.

This is because there may be (on different channels) the same title playing, and doing simply a search would/could cause false positives.

So my idea - yes, sorry I didn't mention this at the start - would be to send the channel and show and look for a match.

I'll see if I can work out a way with what you posted, but as said: I can't see how/where the channel is mentioned.

UnborN · 13 January 2024 23:10

Some modifications to the flow to include the Channel in the result ..
that proved to be a little more difficult since it involved selecting the html elements higher in the chain and then splitting and re-joining the data.

[{"id":"305cb71031a34412","type":"inject","z":"54efb553244c241f","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":85,"y":4140,"wires":[["80a7a0c01b20e948"]],"l":false},{"id":"80a7a0c01b20e948","type":"http request","z":"54efb553244c241f","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://www.foxtel.com.au/now/tv-guide/grid","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[{"keyType":"Accept","keyValue":"","valueType":"text/html","valueValue":""},{"keyType":"User-Agent","keyValue":"","valueType":"other","valueValue":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0"}],"x":190,"y":4140,"wires":[["86a775c6a626657e"]]},{"id":"dd139d12ce4a965e","type":"debug","z":"54efb553244c241f","name":"debug 62","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1240,"y":4140,"wires":[]},{"id":"d69cf8bac35572c2","type":"function","z":"54efb553244c241f","name":"function 3","func":"let movies = []\n\n\nmsg.payload = msg.payload.channel.forEach((channel, index) => {\n    msg.payload.movie[index].forEach((m, i) => {\n        movies.push(\n            {\n                channel: channel[0].trim(),\n                title: m.trim(),\n                time: msg.payload.time[index][i]\n            }\n        )\n    })\n\n})\n\nmsg.payload = movies;\n\nreturn msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":1100,"y":4140,"wires":[["dd139d12ce4a965e"]]},{"id":"86a775c6a626657e","type":"html","z":"54efb553244c241f","name":"channel","property":"payload","outproperty":"payload","tag":".epg-programs-row","ret":"html","as":"single","x":340,"y":4140,"wires":[["261302ca2d51c682"]]},{"id":"a338de30d56a6167","type":"html","z":"54efb553244c241f","name":"movie","property":"payload","outproperty":"payload","tag":".epg-cell .epg-program-name","ret":"text","as":"single","x":630,"y":4140,"wires":[["98b453253a0df79f"]]},{"id":"32fbf595c01e8a7d","type":"html","z":"54efb553244c241f","name":"time","property":"payload","outproperty":"payload","tag":" .epg-cell .sr-only","ret":"html","as":"single","x":630,"y":4200,"wires":[["cbe42ad82ca1af29"]]},{"id":"16cf57328abd411e","type":"html","z":"54efb553244c241f","name":"channel","property":"payload","outproperty":"payload","tag":".epg-channel-name","ret":"text","as":"single","x":620,"y":4080,"wires":[["a48510e9609f2608"]]},{"id":"7b253b1f7547f38d","type":"join","z":"54efb553244c241f","name":"","mode":"auto","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"3","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":810,"y":4080,"wires":[["5ab878cb6514be19"]]},{"id":"a48510e9609f2608","type":"change","z":"54efb553244c241f","name":"","rules":[{"t":"set","p":"topic","pt":"msg","to":"channel","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":715,"y":4080,"wires":[["7b253b1f7547f38d"]],"l":false},{"id":"98b453253a0df79f","type":"change","z":"54efb553244c241f","name":"","rules":[{"t":"set","p":"topic","pt":"msg","to":"movie","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":715,"y":4140,"wires":[["b16177359b9aeeeb"]],"l":false},{"id":"cbe42ad82ca1af29","type":"change","z":"54efb553244c241f","name":"","rules":[{"t":"set","p":"topic","pt":"msg","to":"time","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":715,"y":4200,"wires":[["75c49c55136a95e9"]],"l":false},{"id":"261302ca2d51c682","type":"split","z":"54efb553244c241f","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":470,"y":4140,"wires":[["16cf57328abd411e","a338de30d56a6167","32fbf595c01e8a7d"]]},{"id":"b16177359b9aeeeb","type":"join","z":"54efb553244c241f","name":"","mode":"auto","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"3","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":810,"y":4140,"wires":[["5ab878cb6514be19"]]},{"id":"75c49c55136a95e9","type":"join","z":"54efb553244c241f","name":"","mode":"auto","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"3","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":810,"y":4200,"wires":[["5ab878cb6514be19"]]},{"id":"5ab878cb6514be19","type":"join","z":"54efb553244c241f","name":"","mode":"custom","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"3","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":950,"y":4140,"wires":[["d69cf8bac35572c2"]]}]

I hope its in more workable format

Trying_to_learn · 14 January 2024 10:26

I am very sorry I am not replying to your replies quicker.

Where I am it is horrible weather and it is a lot of time HUMID like you won't believe.

It is not conducive to doing any work on the computer enjoyable.

I'll give it a go now.

Thanks very much.

Looking good!

Please understand I am asking because I don't know.

Ok, so I get that big array and want to search it for channel x and program y.

I first off filter out the unwanted channels.
That's with a switch node, I'm guessing. Easy way.

I'm working on it now, but it is 21:31 here and I am kind of powering down from today.

Trying_to_learn · 14 January 2024 10:46

BEFORE YOU USE THIS, edit the `what I want to watch` node.
(Press the original `inject` node and pick a show which is about to start and fill in the details in the new `inject` node.)

MODIFIED CODE

This is what I have - EXAMPLE!

[{"id":"ed5076c00dd272c1","type":"inject","z":"0918ee609bf69fc7","name":"Get guide with tis for initial shows/times","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":1250,"y":3660,"wires":[["5e655edb9097fa89"]]},{"id":"5e655edb9097fa89","type":"http request","z":"0918ee609bf69fc7","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://www.foxtel.com.au/now/tv-guide/grid","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[{"keyType":"Accept","keyValue":"","valueType":"text/html","valueValue":""},{"keyType":"User-Agent","keyValue":"","valueType":"other","valueValue":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0"}],"x":1170,"y":3830,"wires":[["a63537a59a74e6d8"]]},{"id":"44afecd5a248492c","type":"debug","z":"0918ee609bf69fc7","name":"RAW guide","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":2230,"y":3830,"wires":[]},{"id":"220c62f5fc4e5590","type":"function","z":"0918ee609bf69fc7","name":"function 3","func":"let movies = []\n\n\nmsg.payload = msg.payload.channel.forEach((channel, index) => {\n    msg.payload.movie[index].forEach((m, i) => {\n        movies.push(\n            {\n                channel: channel[0].trim(),\n                title: m.trim(),\n                time: msg.payload.time[index][i]\n            }\n        )\n    })\n\n})\n\nmsg.payload = movies;\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":2060,"y":3830,"wires":[["44afecd5a248492c","8748082553887665"]]},{"id":"a63537a59a74e6d8","type":"html","z":"0918ee609bf69fc7","name":"channel","property":"payload","outproperty":"payload","tag":".epg-programs-row","ret":"html","as":"single","x":1320,"y":3830,"wires":[["457ee776da9ae1d7"]]},{"id":"ad708ba92b036065","type":"html","z":"0918ee609bf69fc7","name":"movie","property":"payload","outproperty":"payload","tag":".epg-cell .epg-program-name","ret":"text","as":"single","x":1610,"y":3830,"wires":[["1a6bf591eac4d83c"]]},{"id":"f8c1d69fa4e024fe","type":"html","z":"0918ee609bf69fc7","name":"time","property":"payload","outproperty":"payload","tag":" .epg-cell .sr-only","ret":"html","as":"single","x":1610,"y":3890,"wires":[["9b51e93122fde748"]]},{"id":"cb31bdf8b963400f","type":"html","z":"0918ee609bf69fc7","name":"channel","property":"payload","outproperty":"payload","tag":".epg-channel-callout-number","ret":"text","as":"single","x":1600,"y":3770,"wires":[["ea1a9f8b643d504e"]]},{"id":"6544f9aafdf56d22","type":"join","z":"0918ee609bf69fc7","name":"","mode":"auto","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"3","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":1790,"y":3770,"wires":[["8610439ee46e499e"]]},{"id":"ea1a9f8b643d504e","type":"change","z":"0918ee609bf69fc7","name":"","rules":[{"t":"set","p":"topic","pt":"msg","to":"channel","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":1695,"y":3770,"wires":[["6544f9aafdf56d22"]],"l":false},{"id":"1a6bf591eac4d83c","type":"change","z":"0918ee609bf69fc7","name":"","rules":[{"t":"set","p":"topic","pt":"msg","to":"movie","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":1695,"y":3830,"wires":[["0b0804130ade5cbd"]],"l":false},{"id":"9b51e93122fde748","type":"change","z":"0918ee609bf69fc7","name":"","rules":[{"t":"set","p":"topic","pt":"msg","to":"time","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":1695,"y":3890,"wires":[["e9f30cc17a2d7ea0"]],"l":false},{"id":"457ee776da9ae1d7","type":"split","z":"0918ee609bf69fc7","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":1450,"y":3830,"wires":[["cb31bdf8b963400f","ad708ba92b036065","f8c1d69fa4e024fe"]]},{"id":"0b0804130ade5cbd","type":"join","z":"0918ee609bf69fc7","name":"","mode":"auto","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"3","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":1790,"y":3830,"wires":[["8610439ee46e499e"]]},{"id":"e9f30cc17a2d7ea0","type":"join","z":"0918ee609bf69fc7","name":"","mode":"auto","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"3","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":1790,"y":3890,"wires":[["8610439ee46e499e"]]},{"id":"8610439ee46e499e","type":"join","z":"0918ee609bf69fc7","name":"","mode":"custom","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"3","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":1930,"y":3830,"wires":[["220c62f5fc4e5590"]]},{"id":"8748082553887665","type":"link out","z":"0918ee609bf69fc7","name":"link out 9","mode":"link","links":["d439893258352a06"],"x":2175,"y":3870,"wires":[]},{"id":"d439893258352a06","type":"link in","z":"0918ee609bf69fc7","name":"link in 6","links":["8748082553887665"],"x":1125,"y":3990,"wires":[["f4bc4ef5ab5fa925"]]},{"id":"5c9d4ab2ac8e1167","type":"inject","z":"0918ee609bf69fc7","name":"What I want to watch","props":[{"p":"payload"},{"p":"channel","v":"101","vt":"str"},{"p":"show","v":"Blue Bloods","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":1190,"y":3720,"wires":[["743d14e4513b57ba"]]},{"id":"743d14e4513b57ba","type":"change","z":"0918ee609bf69fc7","name":"Set flow values","rules":[{"t":"set","p":"channel","pt":"flow","to":"channel","tot":"msg"},{"t":"set","p":"show","pt":"flow","to":"show","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":1420,"y":3720,"wires":[["5e655edb9097fa89"]]},{"id":"f4bc4ef5ab5fa925","type":"switch","z":"0918ee609bf69fc7","name":"Channel","property":"payload[0].channel","propertyType":"msg","rules":[{"t":"eq","v":"channel","vt":"flow"}],"checkall":"true","repair":false,"outputs":1,"x":1260,"y":3990,"wires":[["2ea4ecf231e91d70"]]},{"id":"2ea4ecf231e91d70","type":"switch","z":"0918ee609bf69fc7","name":"Show","property":"payload[2].title","propertyType":"msg","rules":[{"t":"eq","v":"show","vt":"flow"}],"checkall":"true","repair":false,"outputs":1,"x":1430,"y":3990,"wires":[["107b852cf1cd2865","cd9fc6e449301663"]]},{"id":"107b852cf1cd2865","type":"debug","z":"0918ee609bf69fc7","name":"debug 303","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":1620,"y":3990,"wires":[]},{"id":"cd9fc6e449301663","type":"template","z":"0918ee609bf69fc7","name":"","field":"payload","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"{{show}} on {{channel}} at {{payload[2].time}}","output":"str","x":1610,"y":4030,"wires":[["5539c595e0f631fb"]]},{"id":"5539c595e0f631fb","type":"debug","z":"0918ee609bf69fc7","name":"debug 304","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1790,"y":4030,"wires":[]}]

I've tweaked it so now I get the channel NUMBER - not name.
(No offense)

So now I will have to SPLIT the message from your code before it gets to my code (lower part)
Then I parse each message to match that for which I am searching.

Yes?

Trying_to_learn · 14 January 2024 11:16

Slight progress made:

Modified code after initial post.

Made a lot of stupid mistakes.
Left debug stuff in only to show all cards and maybe help explain what I am doing and why.

Replacing the bottom part of the flow I posted with this:

[{"id":"04f2a1a493143134","type":"switch","z":"0918ee609bf69fc7","name":"Wanted channels","property":"payload.channel","propertyType":"msg","rules":[{"t":"lt","v":"450","vt":"num"}],"checkall":"true","repair":false,"outputs":1,"x":1430,"y":3950,"wires":[["c433ca9cb497aae5","c6d7446fcf7bcedc"]]},{"id":"fbfcb21454bacd28","type":"function","z":"0918ee609bf69fc7","name":"Make channel number","func":"const channel = parseInt(msg.payload.channel);\nmsg.payload.channel = channel;\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":1260,"y":4030,"wires":[["04f2a1a493143134"]]},{"id":"c433ca9cb497aae5","type":"switch","z":"0918ee609bf69fc7","name":"Channel","property":"payload.channel","propertyType":"msg","rules":[{"t":"eq","v":"channel","vt":"flow"}],"checkall":"true","repair":false,"outputs":1,"x":1720,"y":3950,"wires":[["dee04885bd4f3641","08d892309e3d4528"]]},{"id":"c6d7446fcf7bcedc","type":"debug","z":"0918ee609bf69fc7","name":"Wanted channels","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":1520,"y":3990,"wires":[]},{"id":"b9255df6b470290f","type":"split","z":"0918ee609bf69fc7","name":"","splt":"","spltType":"str","arraySplt":"1","arraySpltType":"len","stream":true,"addname":"","x":1210,"y":3950,"wires":[["b4547cf2e6f2647a","fbfcb21454bacd28"]]},{"id":"dee04885bd4f3641","type":"switch","z":"0918ee609bf69fc7","name":"Show","property":"payload.title","propertyType":"msg","rules":[{"t":"eq","v":"show","vt":"flow"}],"checkall":"true","repair":false,"outputs":1,"x":1900,"y":3950,"wires":[["64711467d63ae59f","46c41da801d5c919"]]},{"id":"08d892309e3d4528","type":"debug","z":"0918ee609bf69fc7","name":"Matched channel","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":1950,"y":4040,"wires":[]},{"id":"6a28a506d89fd5e5","type":"link in","z":"0918ee609bf69fc7","name":"link in 7","links":["8748082553887665"],"x":1125,"y":3950,"wires":[["b9255df6b470290f"]]},{"id":"b4547cf2e6f2647a","type":"debug","z":"0918ee609bf69fc7","name":"RAW SPLIT","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":1230,"y":3990,"wires":[]},{"id":"64711467d63ae59f","type":"debug","z":"0918ee609bf69fc7","name":"debug 305","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":2060,"y":3950,"wires":[]},{"id":"46c41da801d5c919","type":"template","z":"0918ee609bf69fc7","name":"","field":"payload","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"{{payload.title}} on {{channel}} at {{payload.time}}","output":"str","x":2050,"y":3990,"wires":[["01900ebd14d2f623"]]},{"id":"01900ebd14d2f623","type":"debug","z":"0918ee609bf69fc7","name":"debug 306","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":2230,"y":3990,"wires":[]}]

That seems to work.

Though I would like you to (please) check I haven't made any stupid mistake/s.

But again: You will have to edit the inject node for what I want to watch.

Could those values survive in the message and so not need to be stored a flow variables/context?

UnborN · 14 January 2024 16:20

Nice.

Its up to you .. either using flow Context or using msg.channel and msg.show from the msg that is passed on can work .. it depends on how you tie all this to maybe a dashboard and the rest of your flow.

Trying_to_learn · 15 January 2024 06:14

So it works?

Ok, I have made changed to what I posted, but just wanting to check.

Trying_to_learn · 17 January 2024 00:13

(Only to keep things in context)

I get the time the show starts in {{payload.time}}

So the idea is I send a request to get the more accurate times a good 10 minutes before.
I then have a nominal start time.

~~Say the show is running late.~~ Actually that shouldn't come into the equation.

I get the newly received start time, take off a couple of minutes and then pass that on.

For the sake of making it simpler:

I get msg.payload.time and I want to take off 10 minutes.

Normally that isn't hard.
Just -10 to the minute value.

So, sorry, the brain isn't quite with it.
How can I parse msg.payload.time and subtract 10 minutes from it?

There is another problem that is there too.
If they are running a string of episodes. for the sake of keeping it simple/easy - let's say 30 minute episodes.
I don't want the one NOW SHOWING to be parsed.
So it will need to check that the msg.payload.time isn't more than 20 minutes old.
Given I am saying 10 minutes before projected start time.

I'm not wanting to look like I am dumping this on you.
I'd just like a nudge to work out how to attack it.

I'm guessing something like:
Not sure on the exact syntax.

if `minute` < 10
{
    ((`minute` + `60`) - 10) mod 60
    `hour` = `hour` - 1
}

UnborN · 17 January 2024 05:40

the programs that are now showing have a class .epg-program-on-now
but after all lot of attempts i didnt find a way to filter them out and then traverse up the html elements to select the rest of the information using the HTML node.

i resorted in using the Cheerio library in a Function node, which is the same library the HTML node uses under the hood and Moment to do the time subtraction calculations.

maybe somebody has some idea how to solve it purely with Node-red nodes ..

For now you can try using the following flow that excludes the programs that are already started and also creates a field minus 10min :

[{"id":"cc40a382b78a553d","type":"inject","z":"54efb553244c241f","name":"What I want to watch","props":[{"p":"payload"},{"p":"channel","v":"101","vt":"num"},{"p":"show","v":"Blue Bloods","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":250,"y":4100,"wires":[["ad186a15f513afa2"]]},{"id":"ad186a15f513afa2","type":"http request","z":"54efb553244c241f","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://www.foxtel.com.au/now/tv-guide/grid","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[{"keyType":"Accept","keyValue":"","valueType":"text/html","valueValue":""},{"keyType":"User-Agent","keyValue":"","valueType":"other","valueValue":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0"}],"x":330,"y":4180,"wires":[["b1e6515937ef02c1"]]},{"id":"b1e6515937ef02c1","type":"function","z":"54efb553244c241f","name":"cheerio","func":"const $ = cheerio.load(msg.payload);\nlet programs = $(\".epg-program:not(.epg-program-on-now)\").find(\".epg-program-name\").map(function () {\n  return {\n    show: $(this).text().trim(),\n    channelName: $(this).parent().parent().parent().parent().find(\".epg-channel-name\").text().trim(),\n    channelNumber: parseInt($(this).parent().parent().parent().parent().find(\".epg-channel-number\").text().trim()),\n    startTime: $(this).parent().find(\".sr-only\").text().trim(),\n    alertTime: moment($(this).parent().find(\".sr-only\").text().trim(), \"hh:mma\").subtract(10, 'minutes').format(\"hh:mma\"),\n  }\n}).toArray();\n\nmsg.payload = programs\nreturn msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[{"var":"cheerio","module":"cheerio"},{"var":"moment","module":"moment"}],"x":540,"y":4180,"wires":[["9b5d82b99e932df1"]]},{"id":"9b5d82b99e932df1","type":"debug","z":"54efb553244c241f","name":"debug 310","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":720,"y":4180,"wires":[]}]

Trying_to_learn · 17 January 2024 06:22

Oooh, that's interesting.

The alertTime is handy.

Though startTime or alertTime....

I'm not sure that is good or bad.

Only that ...... Well, ok, I'll have another look and see if I can see how that would work.

Problem detected:
It is looking for Blue Bloods - yes?

Not sure what is going on there.

I'll really have to study that function node code.

This is what I have (slight mod on your original post)

[{"id":"ed5076c00dd272c1","type":"inject","z":"0918ee609bf69fc7","name":"Get guide with tis for initial shows/times","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":1250,"y":3660,"wires":[["5e655edb9097fa89"]]},{"id":"5e655edb9097fa89","type":"http request","z":"0918ee609bf69fc7","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://www.foxtel.com.au/now/tv-guide/grid","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[{"keyType":"Accept","keyValue":"","valueType":"text/html","valueValue":""},{"keyType":"User-Agent","keyValue":"","valueType":"other","valueValue":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0"}],"x":1170,"y":3830,"wires":[["a63537a59a74e6d8"]]},{"id":"44afecd5a248492c","type":"debug","z":"0918ee609bf69fc7","name":"RAW guide","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":2230,"y":3830,"wires":[]},{"id":"220c62f5fc4e5590","type":"function","z":"0918ee609bf69fc7","name":"function 3","func":"let movies = []\n\n\nmsg.payload = msg.payload.channel.forEach((channel, index) => {\n    msg.payload.movie[index].forEach((m, i) => {\n        movies.push(\n            {\n                channel: channel[0].trim(),\n                title: m.trim(),\n                time: msg.payload.time[index][i]\n            }\n        )\n    })\n\n})\n\nmsg.payload = movies;\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":2060,"y":3830,"wires":[["44afecd5a248492c","8748082553887665"]]},{"id":"a63537a59a74e6d8","type":"html","z":"0918ee609bf69fc7","name":"channel","property":"payload","outproperty":"payload","tag":".epg-programs-row","ret":"html","as":"single","x":1320,"y":3830,"wires":[["457ee776da9ae1d7"]]},{"id":"ad708ba92b036065","type":"html","z":"0918ee609bf69fc7","name":"movie","property":"payload","outproperty":"payload","tag":".epg-cell .epg-program-name","ret":"text","as":"single","x":1610,"y":3830,"wires":[["1a6bf591eac4d83c"]]},{"id":"f8c1d69fa4e024fe","type":"html","z":"0918ee609bf69fc7","name":"time","property":"payload","outproperty":"payload","tag":" .epg-cell .sr-only","ret":"html","as":"single","x":1610,"y":3890,"wires":[["9b51e93122fde748"]]},{"id":"cb31bdf8b963400f","type":"html","z":"0918ee609bf69fc7","name":"channel","property":"payload","outproperty":"payload","tag":".epg-channel-callout-number","ret":"text","as":"single","x":1600,"y":3770,"wires":[["ea1a9f8b643d504e"]]},{"id":"6544f9aafdf56d22","type":"join","z":"0918ee609bf69fc7","name":"","mode":"auto","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"3","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":1790,"y":3770,"wires":[["8610439ee46e499e"]]},{"id":"ea1a9f8b643d504e","type":"change","z":"0918ee609bf69fc7","name":"","rules":[{"t":"set","p":"topic","pt":"msg","to":"channel","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":1695,"y":3770,"wires":[["6544f9aafdf56d22"]],"l":false},{"id":"1a6bf591eac4d83c","type":"change","z":"0918ee609bf69fc7","name":"","rules":[{"t":"set","p":"topic","pt":"msg","to":"movie","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":1695,"y":3830,"wires":[["0b0804130ade5cbd"]],"l":false},{"id":"9b51e93122fde748","type":"change","z":"0918ee609bf69fc7","name":"","rules":[{"t":"set","p":"topic","pt":"msg","to":"time","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":1695,"y":3890,"wires":[["e9f30cc17a2d7ea0"]],"l":false},{"id":"457ee776da9ae1d7","type":"split","z":"0918ee609bf69fc7","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":1450,"y":3830,"wires":[["cb31bdf8b963400f","ad708ba92b036065","f8c1d69fa4e024fe"]]},{"id":"0b0804130ade5cbd","type":"join","z":"0918ee609bf69fc7","name":"","mode":"auto","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"3","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":1790,"y":3830,"wires":[["8610439ee46e499e"]]},{"id":"e9f30cc17a2d7ea0","type":"join","z":"0918ee609bf69fc7","name":"","mode":"auto","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"3","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":1790,"y":3890,"wires":[["8610439ee46e499e"]]},{"id":"8610439ee46e499e","type":"join","z":"0918ee609bf69fc7","name":"","mode":"custom","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"3","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":1930,"y":3830,"wires":[["220c62f5fc4e5590"]]},{"id":"8748082553887665","type":"link out","z":"0918ee609bf69fc7","name":"link out 9","mode":"link","links":["d439893258352a06","6a28a506d89fd5e5"],"x":2175,"y":3870,"wires":[]},{"id":"5c9d4ab2ac8e1167","type":"inject","z":"0918ee609bf69fc7","name":"What I want to watch","props":[{"p":"payload"},{"p":"channel","v":"101","vt":"str"},{"p":"show","v":"Law & Order","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":1190,"y":3720,"wires":[["5e655edb9097fa89"]]},{"id":"6a28a506d89fd5e5","type":"link in","z":"0918ee609bf69fc7","name":"link in 7","links":["8748082553887665"],"x":1125,"y":3950,"wires":[["b9255df6b470290f"]]},{"id":"b9255df6b470290f","type":"split","z":"0918ee609bf69fc7","name":"","splt":"","spltType":"str","arraySplt":"1","arraySpltType":"len","stream":true,"addname":"","x":1210,"y":3950,"wires":[["fbfcb21454bacd28"]]},{"id":"64711467d63ae59f","type":"debug","z":"0918ee609bf69fc7","name":"debug 305","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":1810,"y":3950,"wires":[]},{"id":"46c41da801d5c919","type":"template","z":"0918ee609bf69fc7","name":"","field":"payload","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"{{payload.title}} on {{channel}} at {{payload.time}}","output":"str","x":1800,"y":3990,"wires":[["01900ebd14d2f623"]]},{"id":"01900ebd14d2f623","type":"debug","z":"0918ee609bf69fc7","name":"debug 306","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1980,"y":3990,"wires":[]},{"id":"fbfcb21454bacd28","type":"function","z":"0918ee609bf69fc7","name":"Make channel number and filter","func":"const channel = parseInt(msg.payload.channel);\nmsg.payload.channel = channel;\n\nif (msg.payload.channel > 450)\n{\n    return;     //  exit.  Not interested.\n}\nelse\nif (msg.payload.channel < 450)\n{\n    if (msg.payload.channel == msg.channel)\n    {\n        if (msg.payload.title == msg.show)\n        {\n            return msg;\n        }\n        else\n        return;\n    }\n}\n","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":1440,"y":3950,"wires":[["9a1b0213440a7c18"]]},{"id":"9a1b0213440a7c18","type":"junction","z":"0918ee609bf69fc7","x":1620,"y":3950,"wires":[["64711467d63ae59f","46c41da801d5c919"]]}]

UnborN · 17 January 2024 21:53

you can rename those msg properties to more sensible names in the Function node

.. i didnt implement the filtering from your Inject node, as that part was basically the same,
the way you did it in the flow you posted .. you can do either with another Function node or Switch nodes

[{"id":"cc40a382b78a553d","type":"inject","z":"54efb553244c241f","name":"What I want to watch","props":[{"p":"payload"},{"p":"channel","v":"101","vt":"num"},{"p":"show","v":"Blue Bloods","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":130,"y":4100,"wires":[["ad186a15f513afa2"]]},{"id":"ad186a15f513afa2","type":"http request","z":"54efb553244c241f","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://www.foxtel.com.au/now/tv-guide/grid","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[{"keyType":"Accept","keyValue":"","valueType":"text/html","valueValue":""},{"keyType":"User-Agent","keyValue":"","valueType":"other","valueValue":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0"}],"x":190,"y":4180,"wires":[["b1e6515937ef02c1"]]},{"id":"b1e6515937ef02c1","type":"function","z":"54efb553244c241f","name":"cheerio","func":"const $ = cheerio.load(msg.payload);\nlet programs = $(\".epg-program:not(.epg-program-on-now)\").find(\".epg-program-name\").map(function () {\n  return {\n    show: $(this).text().trim(),\n    channelName: $(this).parent().parent().parent().parent().find(\".epg-channel-name\").text().trim(),\n    channelNumber: parseInt($(this).parent().parent().parent().parent().find(\".epg-channel-number\").text().trim()),\n    startTime: $(this).parent().find(\".sr-only\").text().trim(),\n    alertTime: moment($(this).parent().find(\".sr-only\").text().trim(), \"hh:mma\").subtract(10, 'minutes').format(\"hh:mma\"),\n  }\n}).toArray();\n\nmsg = { \"show\": msg.show, \"channel\": msg.channel }\nmsg.payload = programs\nreturn msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[{"var":"cheerio","module":"cheerio"},{"var":"moment","module":"moment"}],"x":340,"y":4180,"wires":[["9b5d82b99e932df1","884d98911ea9ea77"]]},{"id":"9b5d82b99e932df1","type":"debug","z":"54efb553244c241f","name":"RAW guide","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":470,"y":4120,"wires":[]},{"id":"2b731ac5f6380d95","type":"switch","z":"54efb553244c241f","name":"channelNumber < 450","property":"payload.channelNumber","propertyType":"msg","rules":[{"t":"lt","v":"450","vt":"num"}],"checkall":"true","repair":false,"outputs":1,"x":720,"y":4180,"wires":[["af1eacb04e69366a","35b3054d667a4617"]]},{"id":"884d98911ea9ea77","type":"split","z":"54efb553244c241f","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":490,"y":4180,"wires":[["2b731ac5f6380d95"]]},{"id":"af1eacb04e69366a","type":"switch","z":"54efb553244c241f","name":"filter show","property":"payload.show","propertyType":"msg","rules":[{"t":"cont","v":"show","vt":"msg"}],"checkall":"true","repair":false,"outputs":1,"x":920,"y":4180,"wires":[["c11b49bc2997ead7"]]},{"id":"c11b49bc2997ead7","type":"template","z":"54efb553244c241f","name":"","field":"payload","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"{{payload.show}} on {{payload.channelNumber}} at {{payload.startTime}}","output":"str","x":1120,"y":4180,"wires":[["265e18fde0bdc6f2"]]},{"id":"265e18fde0bdc6f2","type":"debug","z":"54efb553244c241f","name":"debug 311","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":1270,"y":4180,"wires":[]},{"id":"35b3054d667a4617","type":"debug","z":"54efb553244c241f","name":"debug 312","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":890,"y":4120,"wires":[]}]

Topic		Replies	Views
Scraping and making sense of a website General http-request	4	222	28 February 2023
Is anyone using XMLTV Feeds? General	0	284	10 July 2019
Trying to retrieve info from a website problems, I'm just a rookie General	19	4609	6 December 2018
How can I scrape Wunderground historic data General	6	662	4 June 2020
Having issues with HTTP-Request General http-request	6	294	3 October 2022