Help with getting information from a site - "scraping"?

General
1

Hi folks.

The John Dory:
I live in Oz and have foxtel.

I also have a flow in NR that automatically changes channels when my favorite show is on.
(Recording isn't a problem as it is all done on their box)

Anyway.....

The EXACT time varies - sometimes, and it is annoying either missing the start (not a big deal, as I can start over) but anyway.....

I was thinking that I set a NOMINAL time (enough before when it will start to not miss the start) and at that time the EPG is parsed and the PUBLISHED time is taken and the time the channel changes is based on that.

I may take off 2 minutes just to be sure. But at this time, that is a long way down the track.

Help - please.
How would I do it.

And before you ask:
No, I've never tried and don't have the URL for the timetable/program yet.
Not much good getting it if I can't use it.

2

Hi,
If you summarize it, what does it give? :sweat_smile:

When I was recording (was a time) I added 15 minutes before and after what the timetable said

3

The time offset is not difficult, but exactly how to get the info does depend on the source, so you need to show that link before anyone can be more specific about how you might scrape the data.

Generally you would use the http request node to get the data and the html node to parse it.

4

You seem to have not understood the question.

I am NOT wanting to record.

I want to get the time table and parse it for the start time, so a couple of minutes before that time the TV changes to that channel.

5

Yeah, ok.

I guess it is a kind of catch 22 scenario

6

This is a link which shows me what I need.

https://www.foxtel.com.au/now/tv-guide/grid

7

I might have a way..... Testing....

8

I think I have it

Let's say I want to get the time of Blue Bloods? the selector will be

$("a[href^=Blue-Bloods] > p.sr-only").first().text()

The above will return "8:50am"

Screenshot 2024-01-12 at 21.36.24

I think you can use the HTML node for it

EDIT
It does look like this website doesn't play nice with the Request node - so your milage may very

9

Im not sure about the accuracy of the html selectors but you can play around with this :

image
image1291Ă—161 14.7 KB

[{"id":"305cb71031a34412","type":"inject","z":"54efb553244c241f","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":140,"y":4180,"wires":[["80a7a0c01b20e948"]]},{"id":"80a7a0c01b20e948","type":"http request","z":"54efb553244c241f","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://www.foxtel.com.au/now/tv-guide/grid","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[{"keyType":"Accept","keyValue":"","valueType":"text/html","valueValue":""},{"keyType":"User-Agent","keyValue":"","valueType":"other","valueValue":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0"}],"x":310,"y":4180,"wires":[["08950add9b13deb4","d80d68b711c5ebfd"]]},{"id":"08950add9b13deb4","type":"html","z":"54efb553244c241f","name":"","property":"payload","outproperty":"payload","tag":".epg-cell .epg-program-name","ret":"text","as":"single","x":590,"y":4120,"wires":[["bfd1390fc9a8d0ba"]]},{"id":"dd139d12ce4a965e","type":"debug","z":"54efb553244c241f","name":"debug 62","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1220,"y":4160,"wires":[]},{"id":"d80d68b711c5ebfd","type":"html","z":"54efb553244c241f","name":"","property":"payload","outproperty":"payload","tag":" .epg-cell .sr-only","ret":"html","as":"single","x":550,"y":4200,"wires":[["474cf6d19ead5ed7"]]},{"id":"c7013900160e3c9e","type":"join","z":"54efb553244c241f","name":"","mode":"custom","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"2","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":890,"y":4160,"wires":[["d69cf8bac35572c2"]]},{"id":"bfd1390fc9a8d0ba","type":"change","z":"54efb553244c241f","name":"","rules":[{"t":"set","p":"topic","pt":"msg","to":"title","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":755,"y":4120,"wires":[["c7013900160e3c9e"]],"l":false},{"id":"474cf6d19ead5ed7","type":"change","z":"54efb553244c241f","name":"","rules":[{"t":"set","p":"topic","pt":"msg","to":"time","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":755,"y":4200,"wires":[["c7013900160e3c9e"]],"l":false},{"id":"d69cf8bac35572c2","type":"function","z":"54efb553244c241f","name":"function 3","func":"msg.payload = msg.payload.title.map( (m,i) => {\n    return { \n        movie: m.trim(),\n        time: msg.payload.time[i]\n    }\n})\n\nreturn msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":1040,"y":4160,"wires":[["dd139d12ce4a965e"]]}]

It does two separate filtrations .. one for the title and one for the time and then merges them into one array

image

.. that you can then search, for a specific "title" to get a matching start time.

2 Likes
10

Looking good.

But I shall admit I am still a bit confused.

I'm guessing this downloads the entire EPG for all channels.
Slowly going through the output.

But I'm not sure how I can get the channel name also.

This is because there may be (on different channels) the same title playing, and doing simply a search would/could cause false positives.

So my idea - yes, sorry I didn't mention this at the start - would be to send the channel and show and look for a match.

I'll see if I can work out a way with what you posted, but as said: I can't see how/where the channel is mentioned.

11

Some modifications to the flow to include the Channel in the result ..
that proved to be a little more difficult since it involved selecting the html elements higher in the chain and then splitting and re-joining the data.

[{"id":"305cb71031a34412","type":"inject","z":"54efb553244c241f","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":85,"y":4140,"wires":[["80a7a0c01b20e948"]],"l":false},{"id":"80a7a0c01b20e948","type":"http request","z":"54efb553244c241f","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://www.foxtel.com.au/now/tv-guide/grid","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[{"keyType":"Accept","keyValue":"","valueType":"text/html","valueValue":""},{"keyType":"User-Agent","keyValue":"","valueType":"other","valueValue":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0"}],"x":190,"y":4140,"wires":[["86a775c6a626657e"]]},{"id":"dd139d12ce4a965e","type":"debug","z":"54efb553244c241f","name":"debug 62","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1240,"y":4140,"wires":[]},{"id":"d69cf8bac35572c2","type":"function","z":"54efb553244c241f","name":"function 3","func":"let movies = []\n\n\nmsg.payload = msg.payload.channel.forEach((channel, index) => {\n    msg.payload.movie[index].forEach((m, i) => {\n        movies.push(\n            {\n                channel: channel[0].trim(),\n                title: m.trim(),\n                time: msg.payload.time[index][i]\n            }\n        )\n    })\n\n})\n\nmsg.payload = movies;\n\nreturn msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":1100,"y":4140,"wires":[["dd139d12ce4a965e"]]},{"id":"86a775c6a626657e","type":"html","z":"54efb553244c241f","name":"channel","property":"payload","outproperty":"payload","tag":".epg-programs-row","ret":"html","as":"single","x":340,"y":4140,"wires":[["261302ca2d51c682"]]},{"id":"a338de30d56a6167","type":"html","z":"54efb553244c241f","name":"movie","property":"payload","outproperty":"payload","tag":".epg-cell .epg-program-name","ret":"text","as":"single","x":630,"y":4140,"wires":[["98b453253a0df79f"]]},{"id":"32fbf595c01e8a7d","type":"html","z":"54efb553244c241f","name":"time","property":"payload","outproperty":"payload","tag":" .epg-cell .sr-only","ret":"html","as":"single","x":630,"y":4200,"wires":[["cbe42ad82ca1af29"]]},{"id":"16cf57328abd411e","type":"html","z":"54efb553244c241f","name":"channel","property":"payload","outproperty":"payload","tag":".epg-channel-name","ret":"text","as":"single","x":620,"y":4080,"wires":[["a48510e9609f2608"]]},{"id":"7b253b1f7547f38d","type":"join","z":"54efb553244c241f","name":"","mode":"auto","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"3","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":810,"y":4080,"wires":[["5ab878cb6514be19"]]},{"id":"a48510e9609f2608","type":"change","z":"54efb553244c241f","name":"","rules":[{"t":"set","p":"topic","pt":"msg","to":"channel","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":715,"y":4080,"wires":[["7b253b1f7547f38d"]],"l":false},{"id":"98b453253a0df79f","type":"change","z":"54efb553244c241f","name":"","rules":[{"t":"set","p":"topic","pt":"msg","to":"movie","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":715,"y":4140,"wires":[["b16177359b9aeeeb"]],"l":false},{"id":"cbe42ad82ca1af29","type":"change","z":"54efb553244c241f","name":"","rules":[{"t":"set","p":"topic","pt":"msg","to":"time","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":715,"y":4200,"wires":[["75c49c55136a95e9"]],"l":false},{"id":"261302ca2d51c682","type":"split","z":"54efb553244c241f","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":470,"y":4140,"wires":[["16cf57328abd411e","a338de30d56a6167","32fbf595c01e8a7d"]]},{"id":"b16177359b9aeeeb","type":"join","z":"54efb553244c241f","name":"","mode":"auto","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"3","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":810,"y":4140,"wires":[["5ab878cb6514be19"]]},{"id":"75c49c55136a95e9","type":"join","z":"54efb553244c241f","name":"","mode":"auto","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"3","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":810,"y":4200,"wires":[["5ab878cb6514be19"]]},{"id":"5ab878cb6514be19","type":"join","z":"54efb553244c241f","name":"","mode":"custom","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"3","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":950,"y":4140,"wires":[["d69cf8bac35572c2"]]}]

I hope its in more workable format

image

2 Likes
12

I am very sorry I am not replying to your replies quicker.

Where I am it is horrible weather and it is a lot of time HUMID like you won't believe.

It is not conducive to doing any work on the computer enjoyable.

I'll give it a go now.

Thanks very much.

Looking good!

Please understand I am asking because I don't know.

Ok, so I get that big array and want to search it for channel x and program y.

I first off filter out the unwanted channels.
That's with a switch node, I'm guessing. Easy way.

I'm working on it now, but it is 21:31 here and I am kind of powering down from today.

13

BEFORE YOU USE THIS, edit the what I want to watch node.
(Press the original inject node and pick a show which is about to start and fill in the details in the new inject node.)

MODIFIED CODE

This is what I have - EXAMPLE!

[{"id":"ed5076c00dd272c1","type":"inject","z":"0918ee609bf69fc7","name":"Get guide with tis for initial shows/times","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":1250,"y":3660,"wires":[["5e655edb9097fa89"]]},{"id":"5e655edb9097fa89","type":"http request","z":"0918ee609bf69fc7","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://www.foxtel.com.au/now/tv-guide/grid","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[{"keyType":"Accept","keyValue":"","valueType":"text/html","valueValue":""},{"keyType":"User-Agent","keyValue":"","valueType":"other","valueValue":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0"}],"x":1170,"y":3830,"wires":[["a63537a59a74e6d8"]]},{"id":"44afecd5a248492c","type":"debug","z":"0918ee609bf69fc7","name":"RAW guide","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":2230,"y":3830,"wires":[]},{"id":"220c62f5fc4e5590","type":"function","z":"0918ee609bf69fc7","name":"function 3","func":"let movies = []\n\n\nmsg.payload = msg.payload.channel.forEach((channel, index) => {\n    msg.payload.movie[index].forEach((m, i) => {\n        movies.push(\n            {\n                channel: channel[0].trim(),\n                title: m.trim(),\n                time: msg.payload.time[index][i]\n            }\n        )\n    })\n\n})\n\nmsg.payload = movies;\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":2060,"y":3830,"wires":[["44afecd5a248492c","8748082553887665"]]},{"id":"a63537a59a74e6d8","type":"html","z":"0918ee609bf69fc7","name":"channel","property":"payload","outproperty":"payload","tag":".epg-programs-row","ret":"html","as":"single","x":1320,"y":3830,"wires":[["457ee776da9ae1d7"]]},{"id":"ad708ba92b036065","type":"html","z":"0918ee609bf69fc7","name":"movie","property":"payload","outproperty":"payload","tag":".epg-cell .epg-program-name","ret":"text","as":"single","x":1610,"y":3830,"wires":[["1a6bf591eac4d83c"]]},{"id":"f8c1d69fa4e024fe","type":"html","z":"0918ee609bf69fc7","name":"time","property":"payload","outproperty":"payload","tag":" .epg-cell .sr-only","ret":"html","as":"single","x":1610,"y":3890,"wires":[["9b51e93122fde748"]]},{"id":"cb31bdf8b963400f","type":"html","z":"0918ee609bf69fc7","name":"channel","property":"payload","outproperty":"payload","tag":".epg-channel-callout-number","ret":"text","as":"single","x":1600,"y":3770,"wires":[["ea1a9f8b643d504e"]]},{"id":"6544f9aafdf56d22","type":"join","z":"0918ee609bf69fc7","name":"","mode":"auto","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"3","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":1790,"y":3770,"wires":[["8610439ee46e499e"]]},{"id":"ea1a9f8b643d504e","type":"change","z":"0918ee609bf69fc7","name":"","rules":[{"t":"set","p":"topic","pt":"msg","to":"channel","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":1695,"y":3770,"wires":[["6544f9aafdf56d22"]],"l":false},{"id":"1a6bf591eac4d83c","type":"change","z":"0918ee609bf69fc7","name":"","rules":[{"t":"set","p":"topic","pt":"msg","to":"movie","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":1695,"y":3830,"wires":[["0b0804130ade5cbd"]],"l":false},{"id":"9b51e93122fde748","type":"change","z":"0918ee609bf69fc7","name":"","rules":[{"t":"set","p":"topic","pt":"msg","to":"time","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":1695,"y":3890,"wires":[["e9f30cc17a2d7ea0"]],"l":false},{"id":"457ee776da9ae1d7","type":"split","z":"0918ee609bf69fc7","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":1450,"y":3830,"wires":[["cb31bdf8b963400f","ad708ba92b036065","f8c1d69fa4e024fe"]]},{"id":"0b0804130ade5cbd","type":"join","z":"0918ee609bf69fc7","name":"","mode":"auto","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"3","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":1790,"y":3830,"wires":[["8610439ee46e499e"]]},{"id":"e9f30cc17a2d7ea0","type":"join","z":"0918ee609bf69fc7","name":"","mode":"auto","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"3","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":1790,"y":3890,"wires":[["8610439ee46e499e"]]},{"id":"8610439ee46e499e","type":"join","z":"0918ee609bf69fc7","name":"","mode":"custom","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"3","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":1930,"y":3830,"wires":[["220c62f5fc4e5590"]]},{"id":"8748082553887665","type":"link out","z":"0918ee609bf69fc7","name":"link out 9","mode":"link","links":["d439893258352a06"],"x":2175,"y":3870,"wires":[]},{"id":"d439893258352a06","type":"link in","z":"0918ee609bf69fc7","name":"link in 6","links":["8748082553887665"],"x":1125,"y":3990,"wires":[["f4bc4ef5ab5fa925"]]},{"id":"5c9d4ab2ac8e1167","type":"inject","z":"0918ee609bf69fc7","name":"What I want to watch","props":[{"p":"payload"},{"p":"channel","v":"101","vt":"str"},{"p":"show","v":"Blue Bloods","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":1190,"y":3720,"wires":[["743d14e4513b57ba"]]},{"id":"743d14e4513b57ba","type":"change","z":"0918ee609bf69fc7","name":"Set flow values","rules":[{"t":"set","p":"channel","pt":"flow","to":"channel","tot":"msg"},{"t":"set","p":"show","pt":"flow","to":"show","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":1420,"y":3720,"wires":[["5e655edb9097fa89"]]},{"id":"f4bc4ef5ab5fa925","type":"switch","z":"0918ee609bf69fc7","name":"Channel","property":"payload[0].channel","propertyType":"msg","rules":[{"t":"eq","v":"channel","vt":"flow"}],"checkall":"true","repair":false,"outputs":1,"x":1260,"y":3990,"wires":[["2ea4ecf231e91d70"]]},{"id":"2ea4ecf231e91d70","type":"switch","z":"0918ee609bf69fc7","name":"Show","property":"payload[2].title","propertyType":"msg","rules":[{"t":"eq","v":"show","vt":"flow"}],"checkall":"true","repair":false,"outputs":1,"x":1430,"y":3990,"wires":[["107b852cf1cd2865","cd9fc6e449301663"]]},{"id":"107b852cf1cd2865","type":"debug","z":"0918ee609bf69fc7","name":"debug 303","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":1620,"y":3990,"wires":[]},{"id":"cd9fc6e449301663","type":"template","z":"0918ee609bf69fc7","name":"","field":"payload","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"{{show}} on {{channel}} at {{payload[2].time}}","output":"str","x":1610,"y":4030,"wires":[["5539c595e0f631fb"]]},{"id":"5539c595e0f631fb","type":"debug","z":"0918ee609bf69fc7","name":"debug 304","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1790,"y":4030,"wires":[]}]

I've tweaked it so now I get the channel NUMBER - not name.
(No offense)

So now I will have to SPLIT the message from your code before it gets to my code (lower part)
Then I parse each message to match that for which I am searching.

Yes?

14

Slight progress made:

Modified code after initial post.

Made a lot of stupid mistakes.
Left debug stuff in only to show all cards and maybe help explain what I am doing and why.

Replacing the bottom part of the flow I posted with this:

[{"id":"04f2a1a493143134","type":"switch","z":"0918ee609bf69fc7","name":"Wanted channels","property":"payload.channel","propertyType":"msg","rules":[{"t":"lt","v":"450","vt":"num"}],"checkall":"true","repair":false,"outputs":1,"x":1430,"y":3950,"wires":[["c433ca9cb497aae5","c6d7446fcf7bcedc"]]},{"id":"fbfcb21454bacd28","type":"function","z":"0918ee609bf69fc7","name":"Make channel number","func":"const channel = parseInt(msg.payload.channel);\nmsg.payload.channel = channel;\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":1260,"y":4030,"wires":[["04f2a1a493143134"]]},{"id":"c433ca9cb497aae5","type":"switch","z":"0918ee609bf69fc7","name":"Channel","property":"payload.channel","propertyType":"msg","rules":[{"t":"eq","v":"channel","vt":"flow"}],"checkall":"true","repair":false,"outputs":1,"x":1720,"y":3950,"wires":[["dee04885bd4f3641","08d892309e3d4528"]]},{"id":"c6d7446fcf7bcedc","type":"debug","z":"0918ee609bf69fc7","name":"Wanted channels","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":1520,"y":3990,"wires":[]},{"id":"b9255df6b470290f","type":"split","z":"0918ee609bf69fc7","name":"","splt":"","spltType":"str","arraySplt":"1","arraySpltType":"len","stream":true,"addname":"","x":1210,"y":3950,"wires":[["b4547cf2e6f2647a","fbfcb21454bacd28"]]},{"id":"dee04885bd4f3641","type":"switch","z":"0918ee609bf69fc7","name":"Show","property":"payload.title","propertyType":"msg","rules":[{"t":"eq","v":"show","vt":"flow"}],"checkall":"true","repair":false,"outputs":1,"x":1900,"y":3950,"wires":[["64711467d63ae59f","46c41da801d5c919"]]},{"id":"08d892309e3d4528","type":"debug","z":"0918ee609bf69fc7","name":"Matched channel","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":1950,"y":4040,"wires":[]},{"id":"6a28a506d89fd5e5","type":"link in","z":"0918ee609bf69fc7","name":"link in 7","links":["8748082553887665"],"x":1125,"y":3950,"wires":[["b9255df6b470290f"]]},{"id":"b4547cf2e6f2647a","type":"debug","z":"0918ee609bf69fc7","name":"RAW SPLIT","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":1230,"y":3990,"wires":[]},{"id":"64711467d63ae59f","type":"debug","z":"0918ee609bf69fc7","name":"debug 305","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":2060,"y":3950,"wires":[]},{"id":"46c41da801d5c919","type":"template","z":"0918ee609bf69fc7","name":"","field":"payload","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"{{payload.title}} on {{channel}} at {{payload.time}}","output":"str","x":2050,"y":3990,"wires":[["01900ebd14d2f623"]]},{"id":"01900ebd14d2f623","type":"debug","z":"0918ee609bf69fc7","name":"debug 306","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":2230,"y":3990,"wires":[]}]

That seems to work.

Though I would like you to (please) check I haven't made any stupid mistake/s.

But again: You will have to edit the inject node for what I want to watch.

Could those values survive in the message and so not need to be stored a flow variables/context?

1 Like
15

Nice.

Its up to you .. either using flow Context or using msg.channel and msg.show from the msg that is passed on can work .. it depends on how you tie all this to maybe a dashboard and the rest of your flow.

image

16

So it works?

Ok, I have made changed to what I posted, but just wanting to check.

:wink:

17

(Only to keep things in context)

I get the time the show starts in {{payload.time}}

So the idea is I send a request to get the more accurate times a good 10 minutes before.
I then have a nominal start time.

Say the show is running late. Actually that shouldn't come into the equation.

I get the newly received start time, take off a couple of minutes and then pass that on.

For the sake of making it simpler:

I get msg.payload.time and I want to take off 10 minutes.

Normally that isn't hard.
Just -10 to the minute value.

So, sorry, the brain isn't quite with it.
How can I parse msg.payload.time and subtract 10 minutes from it?

There is another problem that is there too.
If they are running a string of episodes. for the sake of keeping it simple/easy - let's say 30 minute episodes.
I don't want the one NOW SHOWING to be parsed.
So it will need to check that the msg.payload.time isn't more than 20 minutes old.
Given I am saying 10 minutes before projected start time.

I'm not wanting to look like I am dumping this on you.
I'd just like a nudge to work out how to attack it.

I'm guessing something like:
Not sure on the exact syntax.

if `minute` < 10
{
    ((`minute` + `60`) - 10) mod 60
    `hour` = `hour` - 1
}
18

the programs that are now showing have a class .epg-program-on-now
but after all lot of attempts i didnt find a way to filter them out and then traverse up the html elements to select the rest of the information using the HTML node.

i resorted in using the Cheerio library in a Function node, which is the same library the HTML node uses under the hood and Moment to do the time subtraction calculations.

maybe somebody has some idea how to solve it purely with Node-red nodes ..

For now you can try using the following flow that excludes the programs that are already started and also creates a field minus 10min :

image

[{"id":"cc40a382b78a553d","type":"inject","z":"54efb553244c241f","name":"What I want to watch","props":[{"p":"payload"},{"p":"channel","v":"101","vt":"num"},{"p":"show","v":"Blue Bloods","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":250,"y":4100,"wires":[["ad186a15f513afa2"]]},{"id":"ad186a15f513afa2","type":"http request","z":"54efb553244c241f","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://www.foxtel.com.au/now/tv-guide/grid","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[{"keyType":"Accept","keyValue":"","valueType":"text/html","valueValue":""},{"keyType":"User-Agent","keyValue":"","valueType":"other","valueValue":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0"}],"x":330,"y":4180,"wires":[["b1e6515937ef02c1"]]},{"id":"b1e6515937ef02c1","type":"function","z":"54efb553244c241f","name":"cheerio","func":"const $ = cheerio.load(msg.payload);\nlet programs = $(\".epg-program:not(.epg-program-on-now)\").find(\".epg-program-name\").map(function () {\n  return {\n    show: $(this).text().trim(),\n    channelName: $(this).parent().parent().parent().parent().find(\".epg-channel-name\").text().trim(),\n    channelNumber: parseInt($(this).parent().parent().parent().parent().find(\".epg-channel-number\").text().trim()),\n    startTime: $(this).parent().find(\".sr-only\").text().trim(),\n    alertTime: moment($(this).parent().find(\".sr-only\").text().trim(), \"hh:mma\").subtract(10, 'minutes').format(\"hh:mma\"),\n  }\n}).toArray();\n\nmsg.payload = programs\nreturn msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[{"var":"cheerio","module":"cheerio"},{"var":"moment","module":"moment"}],"x":540,"y":4180,"wires":[["9b5d82b99e932df1"]]},{"id":"9b5d82b99e932df1","type":"debug","z":"54efb553244c241f","name":"debug 310","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":720,"y":4180,"wires":[]}]
1 Like
19

Oooh, that's interesting.

The alertTime is handy.

Though startTime or alertTime....

I'm not sure that is good or bad. :wink:

Only that ...... Well, ok, I'll have another look and see if I can see how that would work.

Problem detected:
It is looking for Blue Bloods - yes?

Screenshot from 2024-01-17 17-18-00
Screenshot from 2024-01-17 17-18-001920Ă—1053 294 KB

Not sure what is going on there.

I'll really have to study that function node code.

This is what I have (slight mod on your original post)

[{"id":"ed5076c00dd272c1","type":"inject","z":"0918ee609bf69fc7","name":"Get guide with tis for initial shows/times","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":1250,"y":3660,"wires":[["5e655edb9097fa89"]]},{"id":"5e655edb9097fa89","type":"http request","z":"0918ee609bf69fc7","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://www.foxtel.com.au/now/tv-guide/grid","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[{"keyType":"Accept","keyValue":"","valueType":"text/html","valueValue":""},{"keyType":"User-Agent","keyValue":"","valueType":"other","valueValue":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0"}],"x":1170,"y":3830,"wires":[["a63537a59a74e6d8"]]},{"id":"44afecd5a248492c","type":"debug","z":"0918ee609bf69fc7","name":"RAW guide","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":2230,"y":3830,"wires":[]},{"id":"220c62f5fc4e5590","type":"function","z":"0918ee609bf69fc7","name":"function 3","func":"let movies = []\n\n\nmsg.payload = msg.payload.channel.forEach((channel, index) => {\n    msg.payload.movie[index].forEach((m, i) => {\n        movies.push(\n            {\n                channel: channel[0].trim(),\n                title: m.trim(),\n                time: msg.payload.time[index][i]\n            }\n        )\n    })\n\n})\n\nmsg.payload = movies;\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":2060,"y":3830,"wires":[["44afecd5a248492c","8748082553887665"]]},{"id":"a63537a59a74e6d8","type":"html","z":"0918ee609bf69fc7","name":"channel","property":"payload","outproperty":"payload","tag":".epg-programs-row","ret":"html","as":"single","x":1320,"y":3830,"wires":[["457ee776da9ae1d7"]]},{"id":"ad708ba92b036065","type":"html","z":"0918ee609bf69fc7","name":"movie","property":"payload","outproperty":"payload","tag":".epg-cell .epg-program-name","ret":"text","as":"single","x":1610,"y":3830,"wires":[["1a6bf591eac4d83c"]]},{"id":"f8c1d69fa4e024fe","type":"html","z":"0918ee609bf69fc7","name":"time","property":"payload","outproperty":"payload","tag":" .epg-cell .sr-only","ret":"html","as":"single","x":1610,"y":3890,"wires":[["9b51e93122fde748"]]},{"id":"cb31bdf8b963400f","type":"html","z":"0918ee609bf69fc7","name":"channel","property":"payload","outproperty":"payload","tag":".epg-channel-callout-number","ret":"text","as":"single","x":1600,"y":3770,"wires":[["ea1a9f8b643d504e"]]},{"id":"6544f9aafdf56d22","type":"join","z":"0918ee609bf69fc7","name":"","mode":"auto","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"3","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":1790,"y":3770,"wires":[["8610439ee46e499e"]]},{"id":"ea1a9f8b643d504e","type":"change","z":"0918ee609bf69fc7","name":"","rules":[{"t":"set","p":"topic","pt":"msg","to":"channel","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":1695,"y":3770,"wires":[["6544f9aafdf56d22"]],"l":false},{"id":"1a6bf591eac4d83c","type":"change","z":"0918ee609bf69fc7","name":"","rules":[{"t":"set","p":"topic","pt":"msg","to":"movie","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":1695,"y":3830,"wires":[["0b0804130ade5cbd"]],"l":false},{"id":"9b51e93122fde748","type":"change","z":"0918ee609bf69fc7","name":"","rules":[{"t":"set","p":"topic","pt":"msg","to":"time","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":1695,"y":3890,"wires":[["e9f30cc17a2d7ea0"]],"l":false},{"id":"457ee776da9ae1d7","type":"split","z":"0918ee609bf69fc7","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":1450,"y":3830,"wires":[["cb31bdf8b963400f","ad708ba92b036065","f8c1d69fa4e024fe"]]},{"id":"0b0804130ade5cbd","type":"join","z":"0918ee609bf69fc7","name":"","mode":"auto","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"3","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":1790,"y":3830,"wires":[["8610439ee46e499e"]]},{"id":"e9f30cc17a2d7ea0","type":"join","z":"0918ee609bf69fc7","name":"","mode":"auto","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"3","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":1790,"y":3890,"wires":[["8610439ee46e499e"]]},{"id":"8610439ee46e499e","type":"join","z":"0918ee609bf69fc7","name":"","mode":"custom","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"3","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":1930,"y":3830,"wires":[["220c62f5fc4e5590"]]},{"id":"8748082553887665","type":"link out","z":"0918ee609bf69fc7","name":"link out 9","mode":"link","links":["d439893258352a06","6a28a506d89fd5e5"],"x":2175,"y":3870,"wires":[]},{"id":"5c9d4ab2ac8e1167","type":"inject","z":"0918ee609bf69fc7","name":"What I want to watch","props":[{"p":"payload"},{"p":"channel","v":"101","vt":"str"},{"p":"show","v":"Law & Order","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":1190,"y":3720,"wires":[["5e655edb9097fa89"]]},{"id":"6a28a506d89fd5e5","type":"link in","z":"0918ee609bf69fc7","name":"link in 7","links":["8748082553887665"],"x":1125,"y":3950,"wires":[["b9255df6b470290f"]]},{"id":"b9255df6b470290f","type":"split","z":"0918ee609bf69fc7","name":"","splt":"","spltType":"str","arraySplt":"1","arraySpltType":"len","stream":true,"addname":"","x":1210,"y":3950,"wires":[["fbfcb21454bacd28"]]},{"id":"64711467d63ae59f","type":"debug","z":"0918ee609bf69fc7","name":"debug 305","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":1810,"y":3950,"wires":[]},{"id":"46c41da801d5c919","type":"template","z":"0918ee609bf69fc7","name":"","field":"payload","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"{{payload.title}} on {{channel}} at {{payload.time}}","output":"str","x":1800,"y":3990,"wires":[["01900ebd14d2f623"]]},{"id":"01900ebd14d2f623","type":"debug","z":"0918ee609bf69fc7","name":"debug 306","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1980,"y":3990,"wires":[]},{"id":"fbfcb21454bacd28","type":"function","z":"0918ee609bf69fc7","name":"Make channel number and filter","func":"const channel = parseInt(msg.payload.channel);\nmsg.payload.channel = channel;\n\nif (msg.payload.channel > 450)\n{\n    return;     //  exit.  Not interested.\n}\nelse\nif (msg.payload.channel < 450)\n{\n    if (msg.payload.channel == msg.channel)\n    {\n        if (msg.payload.title == msg.show)\n        {\n            return msg;\n        }\n        else\n        return;\n    }\n}\n","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":1440,"y":3950,"wires":[["9a1b0213440a7c18"]]},{"id":"9a1b0213440a7c18","type":"junction","z":"0918ee609bf69fc7","x":1620,"y":3950,"wires":[["64711467d63ae59f","46c41da801d5c919"]]}]
20

you can rename those msg properties to more sensible names in the Function node

image

.. i didnt implement the filtering from your Inject node, as that part was basically the same,
the way you did it in the flow you posted .. you can do either with another Function node or Switch nodes

[{"id":"cc40a382b78a553d","type":"inject","z":"54efb553244c241f","name":"What I want to watch","props":[{"p":"payload"},{"p":"channel","v":"101","vt":"num"},{"p":"show","v":"Blue Bloods","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":130,"y":4100,"wires":[["ad186a15f513afa2"]]},{"id":"ad186a15f513afa2","type":"http request","z":"54efb553244c241f","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://www.foxtel.com.au/now/tv-guide/grid","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[{"keyType":"Accept","keyValue":"","valueType":"text/html","valueValue":""},{"keyType":"User-Agent","keyValue":"","valueType":"other","valueValue":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0"}],"x":190,"y":4180,"wires":[["b1e6515937ef02c1"]]},{"id":"b1e6515937ef02c1","type":"function","z":"54efb553244c241f","name":"cheerio","func":"const $ = cheerio.load(msg.payload);\nlet programs = $(\".epg-program:not(.epg-program-on-now)\").find(\".epg-program-name\").map(function () {\n  return {\n    show: $(this).text().trim(),\n    channelName: $(this).parent().parent().parent().parent().find(\".epg-channel-name\").text().trim(),\n    channelNumber: parseInt($(this).parent().parent().parent().parent().find(\".epg-channel-number\").text().trim()),\n    startTime: $(this).parent().find(\".sr-only\").text().trim(),\n    alertTime: moment($(this).parent().find(\".sr-only\").text().trim(), \"hh:mma\").subtract(10, 'minutes').format(\"hh:mma\"),\n  }\n}).toArray();\n\nmsg = { \"show\": msg.show, \"channel\": msg.channel }\nmsg.payload = programs\nreturn msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[{"var":"cheerio","module":"cheerio"},{"var":"moment","module":"moment"}],"x":340,"y":4180,"wires":[["9b5d82b99e932df1","884d98911ea9ea77"]]},{"id":"9b5d82b99e932df1","type":"debug","z":"54efb553244c241f","name":"RAW guide","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":470,"y":4120,"wires":[]},{"id":"2b731ac5f6380d95","type":"switch","z":"54efb553244c241f","name":"channelNumber < 450","property":"payload.channelNumber","propertyType":"msg","rules":[{"t":"lt","v":"450","vt":"num"}],"checkall":"true","repair":false,"outputs":1,"x":720,"y":4180,"wires":[["af1eacb04e69366a","35b3054d667a4617"]]},{"id":"884d98911ea9ea77","type":"split","z":"54efb553244c241f","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":490,"y":4180,"wires":[["2b731ac5f6380d95"]]},{"id":"af1eacb04e69366a","type":"switch","z":"54efb553244c241f","name":"filter show","property":"payload.show","propertyType":"msg","rules":[{"t":"cont","v":"show","vt":"msg"}],"checkall":"true","repair":false,"outputs":1,"x":920,"y":4180,"wires":[["c11b49bc2997ead7"]]},{"id":"c11b49bc2997ead7","type":"template","z":"54efb553244c241f","name":"","field":"payload","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"{{payload.show}} on {{payload.channelNumber}} at {{payload.startTime}}","output":"str","x":1120,"y":4180,"wires":[["265e18fde0bdc6f2"]]},{"id":"265e18fde0bdc6f2","type":"debug","z":"54efb553244c241f","name":"debug 311","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":1270,"y":4180,"wires":[]},{"id":"35b3054d667a4617","type":"debug","z":"54efb553244c241f","name":"debug 312","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":890,"y":4120,"wires":[]}]