special notice from Schedules Direct: Re - mythfilldatabase

Forum rules
Get Help using the Schedules Direct service. NOTE: application issues are probably better resolved via the application's support methods. If you post here, at least include your application name!

Re: special notice from Schedules Direct: Re - mythfilldata

Postby cheetah » Thu Apr 02, 2015 8:22 am

The script given doesn't find any matches, even when correcting for the different log filenames I have locally. Nor do I have any messages from mythfilldatabase in my main syslog history. Nor can I find anything referencing a "RunWindow" in my mythtv database backups (i.e. the settings tables).

I'm away from home right now, so I can't access the mythtv settings GUI easily.

I do have these settings that I think are the "RunWindow" you're referring to?
MythFillPeriod = 1
MythFillMinHour = 2
MythFillMaxHour = 23

Is the Period (which I'm guessing maybe is in days?) perhaps the problem, and the bit that's causing it to ignore the suggested next run time that is less than 24 hours away?
cheetah
 
Posts: 3
Joined: Fri Aug 17, 2007 8:25 am

Re: special notice from Schedules Direct: Re - mythfilldata

Postby fphillips » Thu Apr 02, 2015 9:04 am

There is no evidence of MythFillPeriod in the code, so it is probably an innocuous old setting that didn't get removed. FWIW I also have that setting set to 1 and my SD data is fine.
fphillips
 
Posts: 12
Joined: Tue May 13, 2014 12:16 pm
Location: Austin. TX

Re: special notice from Schedules Direct: Re - mythfilldata

Postby rmeden » Thu Apr 02, 2015 9:23 am

Yea, we already have a DOS attempt by some clients.... they retry immediately upon failure.. even a password failure or expiration! (in neither case would a retry help). That's why I implemented fail2ban to block them.

Interesting bug report back in May.. I didn't remember seeing it.

If the suggested download isn't considered valid, a random download time should be generated... not just use 2am. At least randomize the minute! :) How is it fixed in 0.28?

So: re 0.27 what you're saying is if my suggested time isn't within the MythTV 0.27 window it is not used and eveyone goes at 2am. What's the default window? I'm pretty sure I have the agent string at that point in the code so I can generate a special random time for 0.27. Just moving off the 0 minute will help a bunch.

Robert
rmeden
SD Board Member
 
Posts: 1505
Joined: Tue Aug 14, 2007 2:31 pm
Location: Cedar Hill, TX

Re: special notice from Schedules Direct: Re - mythfilldata

Postby fphillips » Thu Apr 02, 2015 9:48 am

I'm not sure how the housekeeper task works. Presumably you get different results depending on when the backend starts, if it was a scheduled start, and if it is already awake at the window start.

You can see more info from the forum post I linked to. In my health.txt you can see that it was trying to download at 0200UTC and failing. I always suspected that there was a localtime<->UTC mixup there because of the (coincidentally) 2am (local) window start. Keep in mind this was a full year ago. Once patched, it has been solidly using the suggested times.

I can't say whether the current midnight/2am problem is due to this bug, I just recognized Brians' 2-5 run window and failure. Starting with 0.25, new installs defaulted to 0-23, so should be using the suggested time. Pre-0.25 it was 2-5.
fphillips
 
Posts: 12
Joined: Tue May 13, 2014 12:16 pm
Location: Austin. TX

Re: special notice from Schedules Direct: Re - mythfilldata

Postby mattlach » Thu Apr 02, 2015 12:52 pm

[quote="rmeden"]Yea, there have been lots of reports about this.

Looking at http://www.schedulesdirect.org/getack I always get a good suggested time.

Someone replied via email and said that if the suggested time is less than 24 hours away, MythTV will will wait for 24 hours and then poll.

It's also possibly MythTV is treating the UTC as a local time.

I've asked some folks to check the MythTV code. Waiting for a response.

Robert[/quote]


OK, yeah, I was going to report something similar.

The email suggested that users were consciously setting a cronjob to run mythfilldatabase, but I know that I never had.

Mine is running at midnight too, and I don't recall ever seeing any settings to change this anywhere, but I could be wrong.

It might just be a default MythBuntu setting.
mattlach
 
Posts: 16
Joined: Mon Jun 23, 2014 10:46 am

Re: special notice from Schedules Direct: Re - mythfilldata

Postby BrianH » Thu Apr 02, 2015 7:17 pm

fphillips wrote:@BrianH
@cheetah

There is a bug in 0.27 that causes it to ignore suggested runtimes that are outside the window set, even though the window shouldn't be a consideration in your case. It is fixed for 0.28. In the meantime, you can set your window to 0-23 so it doesn't miss any runtimes.

If you will do me a favor, run the script at viewtopic.php?f=5&t=2541#p7623 and pastebin the contents of /tmp/health, I would appreciate it.



@fphillips

I tried the script that you suggested - no output was generated. I noticed that the script was looking for
mythfilldatabase.201* which does not exist in /var/log/mythtv

the contents of the directory are:

-rw-r--r-- 1 syslog adm 5404543 Apr 2 22:04 mythbackend.log
-rw-r--r-- 1 syslog adm 2947690 Apr 2 21:39 mythcommflag.log
-rw-r--r-- 1 syslog adm 1912818 Apr 2 10:56 mythfrontend.log
-rw-r--r-- 1 syslog adm 10498333 Mar 10 07:35 mythfrontend.log.1
-rw-r--r-- 1 syslog adm 2871730 Apr 2 21:00 mythmetadatalookup.log
-rw-r--r-- 1 syslog adm 778433 Apr 2 21:39 mythpreviewgen.log
-rw-r--r-- 1 syslog adm 10685141 Mar 14 07:53 mythpreviewgen.log.1
-rw-r--r-- 1 syslog adm 64279 Apr 2 09:54 mythtv-setup.log

I grepped through all files and didn't find anything that looked relevant.


I've decided to turn off automatic updates and create a crontab entry as suggested by another poster.
BrianH
 
Posts: 2
Joined: Thu Apr 02, 2015 5:43 am

Re: special notice from Schedules Direct: Re - mythfilldata

Postby rmeden » Thu Apr 02, 2015 8:02 pm

mattlach wrote:The email suggested that users were consciously setting a cronjob to run mythfilldatabase, but I know that I never had.


I never imagined MythTV would have such a bug, so I figured that auto-schedule was not the default option and everyone was setting a cron job.

Hopefully the 24 hour rule will work and I can adjust it on my end. We'll know tomorrow (tonight's download) as I bumped everything by a day yesterday.

Robert
rmeden
SD Board Member
 
Posts: 1505
Joined: Tue Aug 14, 2007 2:31 pm
Location: Cedar Hill, TX

Re: special notice from Schedules Direct: Re - mythfilldata

Postby rmeden » Thu Apr 02, 2015 11:41 pm

Update... I've already passed 3 of the large peak times and loads are very low!

For Wednesday's download I made a quick change to the suggested time to bump it by a day.. hoping that would prevent MythTV 0.27 from downloading at 2am local time. It seems to have happened! I should see downloads increase after 9am central today.

Today I modified the suggested times to generate a time in the "optimal time window" (9am-7pm Central). If that time is later than today, bump things by another day. I realize that will cause MythTV to sometimes skip a day, but that's still better than downloading at midnight. I may later tweak the times to only do that bonus day only for MythTV 0.27.

Robert
rmeden
SD Board Member
 
Posts: 1505
Joined: Tue Aug 14, 2007 2:31 pm
Location: Cedar Hill, TX

Re: special notice from Schedules Direct: Re - mythfilldata

Postby bill6502 » Sat Apr 04, 2015 9:36 am

BrianH wrote:I tried the script that you [@fphillips ] suggested - no output was generated. I noticed that the script was looking for
mythfilldatabase.201* which does not exist in /var/log/mythtv


Too late now, but if users aren't getting MFDB logs, then the cause is likely that
in mythtv-setup, the full path to mythfilldatabase was used (possibly by the
distribution.) To get logs, the program name must be exactly:
mythfilldatabase

ref: https://code.mythtv.org/trac/ticket/12169
bill6502
 
Posts: 16
Joined: Fri Sep 19, 2014 4:52 pm

Re: special notice from Schedules Direct: Re - mythfilldata

Postby rmeden » Sat Apr 04, 2015 3:18 pm

This is so frustrating!!

I'm a developer.. I'm used to making a code change compiling it and testing it. This waiting 2 days for results is agonizing!

The good news is things seem to be working better. Yesterday, there were no peaks as the scheduled day users were programmed to skip that day. A more thought out algorithm also was installed.

Last night, there was a large bump at 9pm Central. No idea what that was. (still about 30% below the previous 2am bumps)
Today, there was a small hump between Midnight Central and 5am Central. This is probably remnants of the 2am bug.
A steady increase starting at 9am Central and then steady. (This is what I want).

Today, folks should be getting a scheduled date using the new algorithm (ensures at least 24 hours between downloads).. so we'll see what happens.. I really need to stop watching it! :)

Robert
rmeden
SD Board Member
 
Posts: 1505
Joined: Tue Aug 14, 2007 2:31 pm
Location: Cedar Hill, TX

PreviousNext

Return to Support

Who is online

Users browsing this forum: No registered users and 5 guests

cron