TMS dropping Data Direct!!! But no reason to panic.

rkulagow
SD Staff
Posts: 917
Joined: Tue Aug 14, 2007 3:15 pm

Re: TMS dropping Data Direct!!! But no reason to panic.

Post by rkulagow »

hobbes1069 wrote:Ok, just finding this thread the other day...

I manage the MythTV packages for Fedora Linux on RPM Fusion. I'd like to understand what I can do from a package/distro point of view to make the transition as smooth as possible.

Can this change be done without any interaction from the end user?

Thanks,
Richard
I guess it depends on your end goal. The more testers we can get for the beta service, the better. If that means packaging then that's great. But right now it's very low-level; existing lineups aren't copied from the XML service to the JSON service, and the client script is under development, which is why I've put a line at the top of the README that says, always run git pull before you start.

bill6502
Posts: 16
Joined: Fri Sep 19, 2014 4:52 pm

Re: TMS dropping Data Direct!!! But no reason to panic.

Post by bill6502 »

I'm running an up to date version of API-20140530 on my test backend and noticed that no icon names are stored in my channel table e.g.

Code: Select all

SELECT chanid,callsign FROM channel WHERE icon != '' ORDER BY chanid;
Empty set (0.00 sec)
Is this to be expected? I'm running 0.28-pre and previously, the correct filename was stored. Not the full path as in < 0.27. All
of my icons are still in ~mythtv/.mythtv/channels directory (with 'old' timestamps.) There are 2 new files: s31044_h3_aa.png
and s83118_h3_aa.png. I don't have either of those XMLTVIDs in my channel table.

2nd observation, if the --timezone= option is used, then the mythfilldatabaseLastRunStart and mythfilldatabaseLastRunStart
entries in mythconverg.settings are also stored in the timezone selected, not UTC. Running with the correct timezone allows
the timestamps in the logs to match the format of all other MythTV logs (excluding the usecs.)

jagsd
Posts: 42
Joined: Sun Sep 09, 2007 10:52 am

Re: TMS dropping Data Direct!!! But no reason to panic.

Post by jagsd »

I only noticed this by accident as I went to add a channel to my lineup. I don't visit the site very often. Perhaps you should send an email to everyone to let them know this is coming. If you already did, I did not get it. I do get payment reminders every year and my email address has not changed.

I will check Developer's Corner (as I saw mentioned in this thread) but a few quick questions.

For mythtv, do you use mysql.txt or config.xml to get the database NAME, USERNAME, and PASSWORD? mythtv does and I depend on this as I use different database names ie not mythconverg. Also, I run old mythtv 0.25.2 backend on some pogoplugs which I am loathe to update mythtv version. I run mythfilldatabase via cron.

Do you use the MYTHCONFDIR env var to find mysql.txt or config.xml? I also depend on this.

Thanks.

bill6502
Posts: 16
Joined: Fri Sep 19, 2014 4:52 pm

Re: TMS dropping Data Direct!!! But no reason to panic.

Post by bill6502 »

jagsd wrote: For mythtv, do you use mysql.txt or config.xml to get the database NAME, USERNAME, and PASSWORD? mythtv does and I depend on this as I use different database names ie not mythconverg. Also, I run old mythtv 0.25.2 backend on some pogoplugs which I am loathe to update mythtv version. I run mythfilldatabase via cron.

Do you use the MYTHCONFDIR env var to find mysql.txt or config.xml? I also depend on this.
I don't see any reference to MYTHCONFDIR in the source. Selecting the NAME, USERNAME, and PASSWORD is done on the command line with --dbname, --dbuser and --dbpassword etc. Both sd-utility.php and mfdb-json.php have a --help option.

rmeden
SD Board Member
Posts: 1527
Joined: Tue Aug 14, 2007 2:31 pm
Location: Cedar Hill, TX
Contact:

Re: TMS dropping Data Direct!!! But no reason to panic.

Post by rmeden »

I'm planning on sending a blast to all members (or just members who checked the "newsletter" option), but really there's nothing for the members to do at the moment!

The developers of the various applications need to come up with a solution for their apps, otherwise there's nothing for the end-users to do but worry. I did send a blast to all project leads.

As mentioned last Saturday, I've started alpha testing my DD direct replacement service. That brought out quite a few issues, most solved. Thanks testers! I'll probably post details on how to use it this weekend for more widespread testing. Without a code change, you can switch to it by editing your hosts file! (or change the hostname in the application)

Robert

tgm4883
Posts: 1
Joined: Fri Sep 26, 2014 8:35 am

Re: TMS dropping Data Direct!!! But no reason to panic.

Post by tgm4883 »

rmeden wrote:I'm planning on sending a blast to all members (or just members who checked the "newsletter" option), but really there's nothing for the members to do at the moment!

The developers of the various applications need to come up with a solution for their apps, otherwise there's nothing for the end-users to do but worry. I did send a blast to all project leads.

As mentioned last Saturday, I've started alpha testing my DD direct replacement service. That brought out quite a few issues, most solved. Thanks testers! I'll probably post details on how to use it this weekend for more widespread testing. Without a code change, you can switch to it by editing your hosts file! (or change the hostname in the application)

Robert
Is the hostname edit just for testing? On November 1st, will the old/original hostname work for people still using the DD direct replacement service? (eg. do people that are using legacy applications need to do anything on their end?) As a Mythbuntu developer, from what it sounds like people won't need to do anything on their end as the DD direct replacement service will continue to support them, then we can work on a larger upgrade/move to the JSON service for the next MythTV release (which would be a much better change as it's a more usual suspected large change time)

pgbennett
Posts: 10
Joined: Sat Sep 27, 2014 2:03 pm

Re: TMS dropping Data Direct!!! But no reason to panic.

Post by pgbennett »

Feedback on new json API

I tested the new json scripts on a development machine that is a copy of my production mythtv, so it already has channels and schedules loaded up to today.

The setup ran smoothly using the instructions in the instructions on the github page.

The first time I ran the mfdb-json.php it failed with a message that there was a channel not in my database, because I had not run the U command. Unfortunately it did not identify which channel was the problem. This raises the question of what happens if there is a new channel in schedules direct. It seems my next daily run of mfdb-json.php will fail. How do we prevent this? Can we make it so the job only downloads for channels you have, or ignores data for channels you do not have?

After running the U command it successfully loaded all of the schedules.

When I ran the U command I got bunches of error messages like this.
PHP Warning: file_put_contents(/home/mythtv/.mythtv/channels/s11069_h3_aa.png): failed to open stream: Permission denied in /home/peter/proj/sd/mfdb-json/sd-utility.php on line 1750
Check permissions: could not write to /home/mythtv/.mythtv/channels
Is this home directory hard coded? Maybe it should use a storage directory or the logged-in user's home directory? Will the frontend try to use this directory? What about remote frontends? Since my frontend does not use the mythtv user it may not find these anyway.
I reran the U command after making that directory writeable by all.

The U command deletes all my channels and re-adds them. This causes a few problems:
1. I had channels 1 and 999 set up as dummy channels that I can select for special purposes such as recording from on-demand. They got deleted. How do I prevent this?
2. I had set up priorities on all the channels so that when shows are on multiple channels they record from the best quality and most reliable channel. The U option is setting all priorities to 0 when re-loading all the channels. How do I prevent this?
3. It has loaded many channels that I have not subscribed: HBO, Showtime and such. I do not want these in the table. Mythtv may try to record something off one of those channels and fail. It also added music choice (which does not work with mythtv) and 10 NBA League pass channels which has no program listings, just the words "NBA League Pass" every hour in the schedule.
4. Sometimes there are errors in the lineup, such as missing channels or wrong channel numbers. I can fix these be editing the channel table. Will the edits be overwritten when the channels are next updated?

Will be have the ability to edit the lineups via the web page as we currently can, and select which channels we want?

neopelago
Posts: 16
Joined: Fri Aug 17, 2007 9:51 am

Re: TMS dropping Data Direct!!! But no reason to panic.

Post by neopelago »

My usual xmltv.exe data-only download this morning failed with the error message "This request requires HTTP authentication."
Has something changed already? Is there a fix for this?

rkulagow
SD Staff
Posts: 917
Joined: Tue Aug 14, 2007 3:15 pm

Re: TMS dropping Data Direct!!! But no reason to panic.

Post by rkulagow »

pgbennett wrote:Feedback on new json API

The first time I ran the mfdb-json.php it failed with a message that there was a channel not in my database, because I had not run the U command. Unfortunately it did not identify which channel was the problem. This raises the question of what happens if there is a new channel in schedules direct. It seems my next daily run of mfdb-json.php will fail. How do we prevent this? Can we make it so the job only downloads for channels you have, or ignores data for channels you do not have?
I would need the timestamp and which server provided you with that response (there are multiple servers behind a loadbalancer) so that I can see what's going on. You can send it as a private message.
pgbennett wrote: After running the U command it successfully loaded all of the schedules.

When I ran the U command I got bunches of error messages like this.
PHP Warning: file_put_contents(/home/mythtv/.mythtv/channels/s11069_h3_aa.png): failed to open stream: Permission denied in /home/peter/proj/sd/mfdb-json/sd-utility.php on line 1750
Check permissions: could not write to /home/mythtv/.mythtv/channels
Is this home directory hard coded? Maybe it should use a storage directory or the logged-in user's home directory? Will the frontend try to use this directory? What about remote frontends? Since my frontend does not use the mythtv user it may not find these anyway.
I reran the U command after making that directory writeable by all.
If you run ./sd-utility.php --help, I have a command:
--logo= Directory where channel logos are stored (Default: /home/mythtv/.mythtv/channels)

I'm just doing what I read here:
from http://www.mythtv.org/wiki/Channel_icons

I can try to add more smarts to try to locate the channel directory, but I'm not looking to create too many exceptions either. Pass the --logo parameter for now while I do some more research.
pgbennett wrote: The U command deletes all my channels and re-adds them. This causes a few problems:
1. I had channels 1 and 999 set up as dummy channels that I can select for special purposes such as recording from on-demand. They got deleted. How do I prevent this?
2. I had set up priorities on all the channels so that when shows are on multiple channels they record from the best quality and most reliable channel. The U option is setting all priorities to 0 when re-loading all the channels. How do I prevent this?
3. It has loaded many channels that I have not subscribed: HBO, Showtime and such. I do not want these in the table. Mythtv may try to record something off one of those channels and fail. It also added music choice (which does not work with mythtv) and 10 NBA League pass channels which has no program listings, just the words "NBA League Pass" every hour in the schedule.
4. Sometimes there are errors in the lineup, such as missing channels or wrong channel numbers. I can fix these be editing the channel table. Will the edits be overwritten when the channels are next updated?
For item 1, right now there's no way to prevent this, because the script assumes that it can do what it needs to with the channel table to ensure that it's kept updated. There doesn't appear to be any "lock" column that I can use, and I'm trying to avoid making changes to MythTV tables. If there were, then I could simply have the grabber use that "lock" column as a filter, and only update channels that aren't locked. I'm open to suggestions.

For item 2, I can store the recpriority (I'm assuming that's the column you're referring to) and then put it back once the update is complete. (Since recpriority is an int(10) in MySQL, that's a pretty huge range. We might be able to set a magic value and use that as an indicator that the channel is "locked", but that may interfere with recordings as well, so I'm leery of doing that.)

For item 3, see below.

For item 4, I'd prefer to get those fixed upstream, so if you have some examples that are still happening to you please open a ticket at the Schedules Direct web page.
pgbennett wrote: Will be have the ability to edit the lineups via the web page as we currently can, and select which channels we want?
No - the server doesn't store any state about the user other than which lineups you have. If you don't want a channel, then mark it as not "visible" in the channel table, and the grabber won't get it. Since this is a text-mode app, it doesn't make sense to ask the user about all 900+ channels in their lineup, so that particular functionality is best performed by either mythweb or the mythbacked web-based setup.

jagsd
Posts: 42
Joined: Sun Sep 09, 2007 10:52 am

Re: TMS dropping Data Direct!!! But no reason to panic.

Post by jagsd »

I run mythbackend and mythfilldatabase on some pogoplugs, low powered, resource limited machines but it works. So I am taking a shot at writing my own grabber in javascript running under node.js. I started last night and have a few questions, observations, and problems to discuss.

I applaud your efforts to support differential downloading.

0) When I first started, I got some HTML error messages that lacked the <!DOCTYPE html> tag and thus did not render properly in Chrome. Not a big deal but you might want to add that as some point.

1) For all "modified" dates, is it safe to assume the dates are monotonically increasing? Thus would it be safe to merely string compare dates to detect a modification rather than convert to dates and compare for order?

2) status command

2a) Why is the value of systemStatus an array of one object? It does not seem like there would be more than one object.

2b) What values can status in systemStatus contain other than "Online"?

2c) From other documentation, it would appear that lastDataUpdate only reflects changes in schedules and not lineups. Is this correct? Is it possible a lineup may change without changing lastDataUpdate?

3) schedules/md5 command

3a) Although the md5 signatures might be usefull for server side processing, from a client perspective, aren't "md5" and "modified" equivalent in the sense that if one changes, the other must necessarily also change? What value is the md5 to the client and if none, why send it?

3b) When I first tried this command, I POSTed for my 71 stationIDs and got a 500 server response. I tracked it down to individual stationIDs that apparently had not been loaded with data yet. Here is the stack trace (with my email login changed) for lineup USA-MA58767-X. stationID 14988 (though all are working today, possibly because my last night requests instigated data loading):
Slim Application Error
The application could not run because of the following error:

Details

Type: ErrorException
Code: 8
Message: Undefined variable: action
File: /var/www/html/functions/func.20140530.php
Line: 737
Trace

#0 /var/www/html/functions/func.20140530.php(737): Slim\Slim::handleErrors(8, 'Undefined varia...', '/var/www/html/f...', 737, Array)
#1 /var/www/html/functions/func.20140530.php(1333): getJSON('282+20140530+md...')
#2 /var/www/html/index.php(477): fetchMD5andLastModifiedForSchedules(Array, 'xxxx@yyyy.com...', '4615')
#3 [internal function]: {closure}()
#4 /var/www/html/vendor/slim/slim/Slim/Route.php(462): call_user_func_array(Object(Closure), Array)
#5 /var/www/html/vendor/slim/slim/Slim/Slim.php(1326): Slim\Route->dispatch()
#6 /var/www/html/vendor/slim/slim/Slim/Middleware/Flash.php(85): Slim\Slim->call()
#7 /var/www/html/vendor/slim/slim/Slim/Middleware/MethodOverride.php(92): Slim\Middleware\Flash->call()
#8 /var/www/html/vendor/slim/slim/Slim/Middleware/PrettyExceptions.php(67): Slim\Middleware\MethodOverride->call()
#9 /var/www/html/vendor/slim/slim/Slim/Slim.php(1271): Slim\Middleware\PrettyExceptions->call()
#10 /var/www/html/index.php(795): Slim\Slim->run()
#11 {main}
Also, sometimes rather than an error, I got {"14988":[{"days":13,"md5":null,"modified":null}]}. Are null's to be returned when there is no data?

Post Reply