special notice from Schedules Direct: Re - mythfilldatabase

Forum rules
Get Help using the Schedules Direct service. NOTE: application issues are probably better resolved via the application's support methods. If you post here, at least include your application name!

Re: special notice from Schedules Direct: Re - mythfilldata

Postby fphillips » Wed Apr 15, 2015 5:35 pm

rmeden wrote:none of those are around 2am UTC (or local time).


Yes, in the file I linked to, from Jan through March of last year until I patched it. It's been using the suggested times solidly since then.
fphillips
 
Posts: 12
Joined: Tue May 13, 2014 12:16 pm
Location: Austin. TX

Re: special notice from Schedules Direct: Re - mythfilldata

Postby rmeden » Tue Apr 28, 2015 11:13 pm

Ok, I think I've found the problem. First a pretty picture of the current situation.

Image

Second, I've created a page link for you to see your download history http://www.schedulesdirect.org/getddlog

The problem appears to be MythTV v0.27 (which is also a large number of members!)

This is the pattern:
  • Day 1 no poll
  • Day 2 2am UTC
  • Day 3 2am Localtime
  • repeats

So question for you guys... what time ranges can I suggest for MythTV 0.27 to accept by default?

Robert
rmeden
SD Board Member
 
Posts: 1512
Joined: Tue Aug 14, 2007 2:31 pm
Location: Cedar Hill, TX

Re: special notice from Schedules Direct: Re - mythfilldata

Postby fphillips » Wed Apr 29, 2015 9:02 am

It's clear to me that for these users (not all 0.27 users!) you should be suggesting a random time between 2am and 5am localtime. This will be within their runwindow and the suggested time will be used. I suggest taking the list of users that are requesting 2:00-2:03 utc/local in a three-day group, and changing their suggested time until their UA is upgraded to 0.28 (#12101 is fixed in 0.28).

I briefly looked at my logs from last year and when I wasn't requesting at 2am local or UTC, it was 4:42, 2:53, etc: suggested times within my runwindow.

I don't know if you can see if some of those users are failing to get data during the spikes. This was the problem I had, my listings were dwindling from failed downloads, presumably due to server load spikes at 2am local/utc. If they are getting failures, the above change should mitigate dwindling listings problem, in addition to lowering your load.
fphillips
 
Posts: 12
Joined: Tue May 13, 2014 12:16 pm
Location: Austin. TX

Re: special notice from Schedules Direct: Re - mythfilldata

Postby rmeden » Wed Apr 29, 2015 12:59 pm

Excellent... how did you come up with the 2-5am localtime?

The server is set up for US/Eastern, but I may be able to guess a timezone from the lineups. (that's all the data server really knows). Worst case, I can do 2-5am Central or Mountain and get a good chuck of it. Heck, even spreading the load off the 00 minute will make a big difference in load.

And yes, I monitor the load and how many times I can't return data due to overloading. It hasn't been much lately, but I've been throwing a *LOT* of AWS resources at the problem during the peak hours.

Robert
rmeden
SD Board Member
 
Posts: 1512
Joined: Tue Aug 14, 2007 2:31 pm
Location: Cedar Hill, TX

Re: special notice from Schedules Direct: Re - mythfilldata

Postby fphillips » Wed Apr 29, 2015 1:42 pm

It goes back to this post: viewtopic.php?f=5&t=2541#p7623
Existing, pre-0.25 installs had a default runwindow of 2-5. This didn't matter for people who had mythfilldatabase set to use the suggested time, as it would run whenever the suggested time was. However, with the housekeeper rewrite and release of 0.27, it would only use the suggested time if it was within the window. This is why I have been suggesting people change their runwindow to 0-23, as a workaround until 0.28 is released with my fix.

So...it's so simple I didn't think of it until now. If people aren't using the suggested times outside their runwindow, only suggest times within it. The reason I think all these 2am'ers have a 2-5 runwindow set is because that is the behavior I had before I patched my system. Behavior that started right after I upgraded to 0.27 in Nov 2013. There were many reports during 1H 2014 of timeouts and dwindling listings, most likely after the mass of people upgraded and started overloading the servers. I think by now many of those people have setup a cronjob (bypassing the bug), reinstalled (0.25+ defaults to 0-23), or changed their runwindow (workaround).

Additionally, to be sure, you could check if these users have requested at random (read: suggested) 2-5 times in the past. 4:36, 2:18, etc.

rmeden wrote:And yes, I monitor the load and how many times I can't return data due to overloading. It hasn't been much lately, but I've been throwing a *LOT* of AWS resources at the problem during the peak hours.


Yep, a non-profit doesn't need to be doing that.
Good luck
fphillips
 
Posts: 12
Joined: Tue May 13, 2014 12:16 pm
Location: Austin. TX

Re: special notice from Schedules Direct: Re - mythfilldata

Postby rmeden » Wed Apr 29, 2015 5:21 pm

When you say 2-5am, do you mean 02:00 -> 04:59 or does it only look at the hour.. so 02:00 -> 05:59?
rmeden
SD Board Member
 
Posts: 1512
Joined: Tue Aug 14, 2007 2:31 pm
Location: Cedar Hill, TX

Re: special notice from Schedules Direct: Re - mythfilldata

Postby fphillips » Wed Apr 29, 2015 6:04 pm

From a quick look at the code, you can go to 5:59, which makes sense given they use 0-23 as an infinite window. Is it going to be a problem compressing all those users into 4 hours?
fphillips
 
Posts: 12
Joined: Tue May 13, 2014 12:16 pm
Location: Austin. TX

Re: special notice from Schedules Direct: Re - mythfilldata

Postby rmeden » Wed Apr 29, 2015 6:14 pm

I don't think it will be a problem.. right now they all download at the same *MINUTE* (well every 3 days).
rmeden
SD Board Member
 
Posts: 1512
Joined: Tue Aug 14, 2007 2:31 pm
Location: Cedar Hill, TX

Re: special notice from Schedules Direct: Re - mythfilldata

Postby rmeden » Fri May 01, 2015 2:01 pm

Preliminary results are promising. Last night *should* have been a "no poll" night, but I saw a wide surge starting a little after the other peaks (suggested time is 3am EDT I think). Not tweaking the code yet, just watching.
rmeden
SD Board Member
 
Posts: 1512
Joined: Tue Aug 14, 2007 2:31 pm
Location: Cedar Hill, TX

Re: special notice from Schedules Direct: Re - mythfilldata

Postby rmeden » Tue May 05, 2015 9:01 pm

Image

Looks to me like it's pretty much done! There is still one peak, but I can live with that. I've already reduced the max server counts and stopped the database copy from growing to a huge server. (saving $$$)

Next I'm going to look at the low-level constant downloads. I've already noticed a few folks downloading many times a day, probably by mistake.

Thanks for everyone's help!

Robert
rmeden
SD Board Member
 
Posts: 1512
Joined: Tue Aug 14, 2007 2:31 pm
Location: Cedar Hill, TX

PreviousNext

Return to Support

Who is online

Users browsing this forum: Google [Bot] and 13 guests

cron