# Help with getting information from a site - "scraping"?
**URL:** https://discourse.nodered.org/t/help-with-getting-information-from-a-site-scraping/84502
**Category:** General
**Created:** [12 January 2024 05:11 UTC](https://discourse.nodered.org/t/help-with-getting-information-from-a-site-scraping/84502 "2024-01-12T05:11:23Z")
**Posts on this page:** 11
**Page:** 2
### Author:  [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)
#### Post date: [18 January 2024 04:04 UTC](https://discourse.nodered.org/t/help-with-getting-information-from-a-site-scraping/84502/21 "2024-01-18T04:04:05Z")
Thanks and sorry I was only showing you what I did.
The names aren't a problem in the _new_ `function` node.
> [@UnborN](#):
>
> you can rename those msg properties to more sensible names in the Function node
>
> 
It is more me not having the understanding of how to read it. The structure and how it works.
I know it has been discussed in this thread - I believe - but it is still just _magic_ how it works.
😉
I'll look at the code you posted now.
So this is what `cheerio` does:
(in picture)
Pretty much - yes?
---
### Author:  [@shrickus](https://discourse.nodered.org/u/shrickus)
#### Post date: [18 January 2024 14:39 UTC](https://discourse.nodered.org/t/help-with-getting-information-from-a-site-scraping/84502/22 "2024-01-18T14:39:00Z")
> [@Trying\_to\_learn](#):
>
> How can I parse `msg.payload.time` and subtract 10 minutes from it?
The moment library is great for things like this... but if you want to stick with just JS code, this should work. The trick is getting the time string into a format that the Date parser recognizes (the guide date + the "locale" time):
```auto
// use today's date, or parse the info from the scraped guide?
let guideDate = new Date().toLocaleDateString();
// we need the time string in '04:30 pm' format
let startTime = msg.payload.replace(/.*(\d{1,2}):(\d{2})\s*([aApPmM]+.*)/, "$1:$2 $3");
// turn the start datetime string into a UTC timestamp
let startDate = new Date(`${guideDate} ${startTime}`);
// subtract 10 min (as milliseconds)
let alertDate = new Date(startDate - (10 * 60 * 1000));
node.log(`Alert time is ${alertTIme.toLocaleTimeString()}`);
```
You now have a Date that can be used to set a notification, or compare to the current timestamp for filtering.
---
### Author:  [@UnborN](https://discourse.nodered.org/u/UnborN)
#### Post date: [18 January 2024 15:19 UTC](https://discourse.nodered.org/t/help-with-getting-information-from-a-site-scraping/84502/23 "2024-01-18T15:19:18Z")
> [@Trying\_to\_learn](#):
>
> Pretty much - yes?
Exactly .. the circled part was replaced with the Function node and only because
the need to filter out the programs that are now playing (`.epg-program-on-now`) and also find their `parent()` channel number etc was so complex to do with just HTML nodes.
---
### Author:  [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)
#### Post date: [18 January 2024 19:52 UTC](https://discourse.nodered.org/t/help-with-getting-information-from-a-site-scraping/84502/24 "2024-01-18T19:52:14Z")
Thanks, but.....
😕
Say I do a _nominal scan_ of the online EPG say 13 minutes before the _usual_ start time.
Alas that may not really be enough but nothing is perfect and there is/are limits to how much latitude I'm going to give Foxtel on changing their start times.
And again: These are rounded values to keep the example simple.
I send into the flow what I want to watch and on which channel.
I get back a message like this:
```auto
{"show":"Bull","channelName":"Foxtel One","channelNumber":101,"startTime":"7:25am","alertTime":"07:15"}
```
And in my example it would be just after `07:10`
A few things have to happen:
## (I digress, this is what happens now:)
The show is _published_ to start at `07:25`.
So at `07:23` an `inject` node sends a message with the channel number (and other stuff more for human readability when debugging) to MORE CODE that then changes the channel on my TV that has HDMI CEC to the Foxtel box and that changes channel.
But if it is running late or early that isn't fun. I could miss the start or have the channel change and I watch more of the previous shown than I would like.
End of story.
So as I was saying:
It is `07:12` NOW, the show is said to start at `07:25`.
So ideally I want the channel to change at 07:23`
The above message is received.
Time NOW is `07:12` I want the channel to change at `07:23`. `11` minutes.
So a `variable delay` node is in the flow and is told to wait `11 minutes`.
But the `11` is VARIABLE.
Determined from the NOW time and 2 minutes before the received `msg.payload.startTime`.
Alas I am thrown by your code as it uses tricks I don't understand.
I'll give it a go and see what happens.
---
### Author:  [@shrickus](https://discourse.nodered.org/u/shrickus)
#### Post date: [18 January 2024 21:50 UTC](https://discourse.nodered.org/t/help-with-getting-information-from-a-site-scraping/84502/25 "2024-01-18T21:50:28Z")
Ah, ok, I see what you mean -- so whenever the `inject` node runs, you need to get the show's start time, subract 2 minutes (is that a fixed value?), and then determine how much time to send to the `variable delay` node?
Much of the sample code I sent would still be usable, but the delay value is calculated like this:
```auto
// use today's date, or parse the info from the scraped guide?
let guideDate = new Date().toLocaleDateString();
// we need the time string in '04:30 pm' format
let startTime = msg.payload.replace(/.*(\d{1,2}):(\d{2})\s*([aApPmM]+.*)/, "$1:$2 $3");
// turn the start datetime string into a UTC timestamp
let startDate = new Date(`${guideDate} ${startTime}`);
// subtract 2 min (as milliseconds) to get the recording timestamp
let recordDate = new Date(startDate - (2 * 60 * 1000));
// subtract the current timestamp from the recording timestamp
let delayMs = recordDate - Date.now();
node.log(`Delay time is ${delayMs} ms.`);
// don't send the delay value if it is less than 0 ms in the future
if (delayMs > 0) {
msg.payload = { delay: delayMs }; // or whatever needs to go in payload
return msg;
}
return null;
```
---
### Author:  [@craigcurtin](https://discourse.nodered.org/u/craigcurtin)
#### Post date: [19 January 2024 00:57 UTC](https://discourse.nodered.org/t/help-with-getting-information-from-a-site-scraping/84502/26 "2024-01-19T00:57:23Z")
Hey Andrew - i guess the bigger question here is what are you really trying to achieve ?
You have Foxtel now - presumably with the IQ box that acts as an NVR for you and will record as required (series etc) its usually pretty good in terms of start and end times etc (i can not believe you are still paying Foxtels exorbitant fees !)
So what are you trying to achieve by sittng in front of the screen and having it change channels for you - are you about to lose the use of your arms for the remote - or is there something more to this ?
Craig
---
### Author:  [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)
#### Post date: [19 January 2024 08:23 UTC](https://discourse.nodered.org/t/help-with-getting-information-from-a-site-scraping/84502/27 "2024-01-19T08:23:15Z")
The "before start time" I take off is nominal - I think is the word.
Just enough time to not miss the start, as the actual start time is not always what happens.
I could make it 1 minute, 90 seconds, or something like that.
Just a _best fit_ value to change channels before the program starts.
So the message on sent from this to the next node (`delay`) MUST have `msg.delay` and some other stuff set.
Sorry I am severe damage control just now and not really with it.
Does that help?
---
### Author:  [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)
#### Post date: [19 January 2024 08:31 UTC](https://discourse.nodered.org/t/help-with-getting-information-from-a-site-scraping/84502/28 "2024-01-19T08:31:34Z")
Hi Craig.
Alas the popups that used to happen don't that allow you to book shows to record, watch etc.
But that is all handled inside the foxtel box when I'm recording stuff.
What I try to do is WATCH (yes, I know: weird concept in this day and age) some shows.
the times they are on are not _booked_ to anything else and so it is nice to watch them then and there rather than record them and have to find time later on.
As I can't control the Foxtel box directly, I have a SMART TV that has HDMI CEC.
So I am listening to one of the music channels - background noise - and a show I want to watch is about to start.
That way I don't have to be sitting there with the remote in my hand watching for that specific time to happen and change the channel.
As is they are very fixed times. Not too bad, but now and then Foxtel decide to move them around a few minutes here and there.
Quite annoying when the show has been moved from (say) 17:10 to 17:30. The channel changes at 17:08 and I have to either put up with that show or flick back to the music and try to remember to change at the new time.
The last part:
No it is just something I played with once and it is handy now and then.
A show is on at (again: EXAMPLE!) channel 101 @ 17:10 then at 17:45 there is another show on channel 108.
The music is playing.
2 minutes before it changes to 101.
I watch that show.
Then at 17:43 it changes to 108 and I watch the next show.
I can't/don't/getting worse at remembering what is on when.
So this changes the channels for me so I can watch the shows I want to on specific days and specific times on specific channels.
This whole thing allows a bit of flexibility with the start times.
---
### Author:  [@dceejay](https://discourse.nodered.org/u/dceejay)
#### Post date: [19 January 2024 08:58 UTC](https://discourse.nodered.org/t/help-with-getting-information-from-a-site-scraping/84502/29 "2024-01-19T08:58:06Z")
> [@Trying\_to\_learn](#):
>
> @ 17:10 then at 17:45 there is another show on...
So you don't mind missing the last few minutes of the first program when it switches over early ?
---
### Author:  [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)
#### Post date: [19 January 2024 08:59 UTC](https://discourse.nodered.org/t/help-with-getting-information-from-a-site-scraping/84502/30 "2024-01-19T08:59:22Z")
Nothing is perfect.
And that is an example.
---
### Author:  [@system](https://discourse.nodered.org/u/system)
#### Post date: [19 March 2024 09:00 UTC](https://discourse.nodered.org/t/help-with-getting-information-from-a-site-scraping/84502/31 "2024-03-19T09:00:00Z")
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.
[Previous page](https://discourse.nodered.org/t/help-with-getting-information-from-a-site-scraping/84502.md?page=1)