special notice from Schedules Direct: Re - mythfilldatabase

Forum rules
Get Help using the Schedules Direct service. NOTE: application issues are probably better resolved via the application's support methods. If you post here, at least include your application name!
fphillips
Posts: 12
Joined: Tue May 13, 2014 12:16 pm
Location: Austin. TX

Re: special notice from Schedules Direct: Re - mythfilldata

Post by fphillips » Sat Apr 04, 2015 7:35 pm

2100CST = 0200UTC
This is the time I was getting repeated failed downloads. Most likely this 9pm bump you saw is the bug I mentioned. This subset of users have a 2am window start and the housekeeper is running mythfilldatabase at 0200UTC instead of 2am local time.

You may want to mail them and see if they meet the criteria:
1. MythTV 0.27
2. Have MythFillGrabberSuggestsTime=True
3. Have a narrow RunWindow set. This is typically those who originally installed preceding the 0.25 release. Starting in 0.25, installs defaulted to an infinite run window (0-23).

If so, ask them to set the window to 0-23 as a workaround until 0.28 is released. This should allow them to use the suggested times.

rmeden
SD Board Member
Posts: 1525
Joined: Tue Aug 14, 2007 2:31 pm
Location: Cedar Hill, TX
Contact:

Re: special notice from Schedules Direct: Re - mythfilldata

Post by rmeden » Fri Apr 10, 2015 7:12 am

Unfortunately, things have gotten worse.. :( The 2am UTC peak has gotten larger and I don't see the 2am local peaks!

I originally tried to suggest an 8 hour time window when the data is freshest and before prime time. Day before yesterday I tried a 22 hour window. (both ensuring over 24 hours has passed)

Today, I'm going to try and code a 22 hour window ensuring next local day.

If that doesn't help, I may switch to a 2 hour window around the local time. It's not great, but anything is better than everyone downloading at the 00 minute!

I've looked at the agent string supplied and I don't see a pattern. :(

I know I can control it somewhat since I was able to get these boxes to skip a day!

ChristopherNeufeld
Posts: 4
Joined: Wed Apr 01, 2015 7:51 pm

Re: special notice from Schedules Direct: Re - mythfilldata

Post by ChristopherNeufeld » Tue Apr 14, 2015 3:07 am

Well, my box has switched back to local midnight retrieves. Up to Monday morning everything was as desired. It filled at 08h30, but then the suggested next fill time was 17h11 same day. So, it ran at midnight this morning instead. Now, the next suggested time is today at 13h42; it won't run then, it'll wait until midnight.

fphillips
Posts: 12
Joined: Tue May 13, 2014 12:16 pm
Location: Austin. TX

Re: special notice from Schedules Direct: Re - mythfilldata

Post by fphillips » Tue Apr 14, 2015 6:24 am

It sounds like you have the run window set to 2-5 and suggested times outside of that are being ignored. Set it to 0-23 in mythtv-setup.

ChristopherNeufeld
Posts: 4
Joined: Wed Apr 01, 2015 7:51 pm

Re: special notice from Schedules Direct: Re - mythfilldata

Post by ChristopherNeufeld » Tue Apr 14, 2015 8:45 am

fphillips wrote:It sounds like you have the run window set to 2-5 and suggested times outside of that are being ignored. Set it to 0-23 in mythtv-setup.
No, my run window has always been set to 0-23. The problem is a recurrence of that issue I described in my first posting in this thread. The Schedules Direct server is sending a suggested next polling time that is in the same localtime calendar day as the previous one. MythTV will not honour such requests, it will always wait until the next localtime calendar day, then run immediately, when that happens. Something changed yesterday in Schedules Direct, in that it sent such a same-day suggestion.

rmeden
SD Board Member
Posts: 1525
Joined: Tue Aug 14, 2007 2:31 pm
Location: Cedar Hill, TX
Contact:

Re: special notice from Schedules Direct: Re - mythfilldata

Post by rmeden » Tue Apr 14, 2015 3:05 pm

Interesting...

Day before yesterday (so yesterday's poll time) I went back to the original suggested time algorithm. I was set to about a 14 hour window at lest 24 hours ahead.

I did that because I saw a noticeable drop in the daily user count and still had huge peaks... they just moved.. from starting at 11pm US/Central to 9am US/Central! I wanted to see if things went back to where they were before.

Interesting that you were seeing good download times... I guess people like you were just being buried in the data and didn't notice.

I also looked through the user agent strings for a pattern of folks who were downloading at various times... there are so many different versions of MythTV out there, but I didn't see a pattern.

I'm thinking about switching to a "guaranteed next local day" download. I'm also thinking of just randomizing the minute of whatever hour they download.

Robert

ChristopherNeufeld
Posts: 4
Joined: Wed Apr 01, 2015 7:51 pm

Re: special notice from Schedules Direct: Re - mythfilldata

Post by ChristopherNeufeld » Tue Apr 14, 2015 5:36 pm

rmeden wrote:Interesting...
I did that because I saw a noticeable drop in the daily user count and still had huge peaks... they just moved.. from starting at 11pm US/Central to 9am US/Central! I wanted to see if things went back to where they were before.

Interesting that you were seeing good download times... I guess people like you were just being buried in the data and didn't notice.
The drop in daily user count is believable, because your suggested next fill times were sometimes two calendar days in the future. Looking at my logs for April and recording completion times:
April 1: 00h04
April 2: 14h57
April 3: NO FILL
April 4: 10h31
April 5: 13h52
April 6: 16h26
April 7: NO FILL
April 8: 16h29
April 9: NO FILL
April 10: 10h36
April 11: 20h02
April 12: NO FILL
April 13: 08h37
April 14: 00h03

That's 9 fills in 13 days, which, if my situation is not unusual, would manifest as noticeably lower daily counts.

rmeden
SD Board Member
Posts: 1525
Joined: Tue Aug 14, 2007 2:31 pm
Location: Cedar Hill, TX
Contact:

Re: special notice from Schedules Direct: Re - mythfilldata

Post by rmeden » Tue Apr 14, 2015 8:17 pm

Ok I just committed a new algorithm.. (which seems so simple.. not sure why I didn't do it at first)
  • Generate a time from 10am->7pm US/Eastern today (after load, before prime time)
  • Bump time by 24 hours
  • convert to UTC and send
Users should be guaranteed a time their next local day, even if they are US/Pacific

What happens if that time is outside the MythTV download window? Does MythTV download at the start of the next download window? Does it do it at 2am?

Robert

(BTW, thanks for the analysis Christopher!)

fphillips
Posts: 12
Joined: Tue May 13, 2014 12:16 pm
Location: Austin. TX

Re: special notice from Schedules Direct: Re - mythfilldata

Post by fphillips » Wed Apr 15, 2015 8:42 am

rmeden wrote:What happens if that time is outside the MythTV download window? Does MythTV download at the start of the next download window? Does it do it at 2am?
All I know is that one year ago when the suggested times were outside my window (2-5) it would try near the top of 0200UTC and fail (presumably due to traffic). You can see that here: download/file.php?id=44

So, did these bumps just start recently or have they been going on a long time and you just noticed? Maybe there is a timezone pattern.

Here are my recent runs:
2015-03-28 15:06
2015-03-29 16:11
2015-03-30 18:06
2015-03-31 17:58
2015-04-01 16:12
2015-04-02 13:12
2015-04-04 16:59
2015-04-06 13:41
2015-04-07 18:08
2015-04-09 10:18
2015-04-10 19:13
2015-04-12 15:07
2015-04-14 06:38
2015-04-14 18:42
2015-04-15 09:23

rmeden
SD Board Member
Posts: 1525
Joined: Tue Aug 14, 2007 2:31 pm
Location: Cedar Hill, TX
Contact:

Re: special notice from Schedules Direct: Re - mythfilldata

Post by rmeden » Wed Apr 15, 2015 5:18 pm

none of those are around 2am UTC (or local time).

The 2am (UTC + Local) surges have been around ever since SD-DD started. I've just been throwing hardware (and money) at the problem. Now I'm trying to optimize things and reduce costs... we know SD-DD works!

Robert

Post Reply