Changes in listing data

Discussion about Schedules Direct grabber code and data formats.
Post Reply
eytwoierfskj
Posts: 6
Joined: Sat Jul 23, 2011 4:43 pm

Changes in listing data

Post by eytwoierfskj »

I was reading one of the other threads here (viewtopic.php?f=8&t=707) (the last 2 posts in the thread) about how many days worth of data should be downloaded at a time.

While I haven't found anything that says so, the implication from that thread is that changes are an ongoing process (which makes sense), and that even limiting your downloads to 1 week doesn't ensure that the listing data won't change the very next day (which also makes sense, but is annoying).

Since I haven't seen anything that could be described as a change notification system, or a revision number for schedule data, I'm assuming that the only way to be sure I am showing most recently available listings is to just keep re-downloading the data.

I seem to recall having read somewhere that SD gets nightly downloads from Tribune (true?). So downloading more than once a day is just wasted bandwidth. Is there a way to detect when SD has its new data? I guess what I'm looking for is just some way to get a UTC datetime of your latest update from Tribune. With that I can tell if I should wipe out my stored 7 days worth of data and get fresh.

Has anyone done any analysis to see how often data actually changes? Perhaps changes inside 7 days (or 5 days, etc) are theoretically possible, but never really happen?

rmeden
SD Board Member
Posts: 1563
Joined: Tue Aug 14, 2007 2:31 pm
Location: Cedar Hill, TX
Contact:

Re: Changes in listing data

Post by rmeden »

The data servers used to be updated once a day, but now they get a few updates throughout the day.

There's nothing wrong with downloading 2 weeks every day, but if you wanted to keep things smaller, I would download today, tomorrow, the next day, day 7, then day 14. (also fill in any holes)

Usually things are pretty stable 7 days out, but there are some updates a day or so ahead. Updates aren't fast enough to cover "immediate" things like sports running late, or even a spur-of-the-moment presidential press conference.

Robert

eytwoierfskj
Posts: 6
Joined: Sat Jul 23, 2011 4:43 pm

Re: Changes in listing data

Post by eytwoierfskj »

Thank you for the prompt and detailed response.

Unfortunately (for me) this answer, while informative, makes my life much more complicated.

If the goal of the project is to select what programs you want to record for the next week, then to determine the most efficient times to record those programs, finding out that the week's schedule changes every day (and several times a day) is not the news you are hoping to hear.

Given how detailed your response was, if there were any (easy) way of discovering what had changed or even when the most recent updates had occurred, I expect you would have mentioned. Which makes things even more challenging.

I'm going to have to ponder this one.

rmeden
SD Board Member
Posts: 1563
Joined: Tue Aug 14, 2007 2:31 pm
Location: Cedar Hill, TX
Contact:

Re: Changes in listing data

Post by rmeden »

Nope, no "last updated" flag.

I wrote tv_check (part of xmltv) ages go to generate a html report of scheduled shows, highlighting things "unexpected" (duration, start-time change, extra episodes, etc) What I did was basically download a full week's data then generate the report based on that.

It sounds like something similar, so I would simply generate a new recording schedule for each run.

Robert

Post Reply