initial program load fails: 500 Internal Server Error

Post Reply
tcordes
Posts: 4
Joined: Sat Jun 06, 2020 1:58 am

initial program load fails: 500 Internal Server Error

Post by tcordes »

I've been forced by mythtv updates to switch to xmltv and tv_grab_zz_sdjson_sqlite grabber. I've got everything setup, but tv_grab_zz_sdjson_sqlite is failing with 500 Internal Server Error if I watch its debug mode output.

It gets to the point where it POSTs all the program IDs it wants in obtainPrograms to https://json.schedulesdirect.org/20141201/programs. It sits there for many minutes not doing anything (strace shows it's retrying the hits after sleeping for 20s).

I toyed around with the code and if I reduce the number of programs tv_grab_zz_sdjson_sqlite requests in obtainPrograms, then the sd servers return ok and I get data!

So I'm thinking there's just too many programs in the POST request and your servers can't handle it before something timesout on your side? I don't know enough about tv_grab_zz_sdjson_sqlite to make it request less programs at a time without screwing things up.

Here's debug from tv_grab_zz_sdjson_sqlite when the error occurs:

Code: Select all

DEBUG:                    "_headers" => bless( {
DEBUG:                                           "title" => "500 Internal Server Error",
DEBUG:                                           "client-date" => "Sat, 06 Jun 2020 09:28:06 GMT",
DEBUG:                                           "content-type" => "text/html",
DEBUG:                                           "content-length" => 202,
DEBUG:                                           "date" => "Sat, 06 Jun 2020 09:28:06 GMT",
DEBUG:                                           "client-response-num" => 1,
DEBUG:                                           "server" => "nginx/1.14.0 (Ubuntu)",
DEBUG:                                           "connection" => "keep-alive",
DEBUG:                                           "::std_case" => {
DEBUG:                                                             "client-ssl-cipher" => "Client-SSL-Cipher",
DEBUG:                                                             "client-ssl-socket-class" => "Client-SSL-Socket-Class",
DEBUG:                                                             "client-ssl-cert-issuer" => "Client-SSL-Cert-Issuer",
DEBUG:                                                             "client-ssl-cert-subject" => "Client-SSL-Cert-Subject",
DEBUG:                                                             "client-peer" => "Client-Peer",
DEBUG:                                                             "title" => "Title",
DEBUG:                                                             "client-date" => "Client-Date",
DEBUG:                                                             "client-response-num" => "Client-Response-Num"
DEBUG:                                                           },
DEBUG:                                           "client-ssl-cert-issuer" => "/C=US/O=Amazon/OU=Server CA 1B/CN=Amazon",
DEBUG:                                           "client-ssl-cert-subject" => "/CN=*.json.schedulesdirect.org",
DEBUG:                                           "client-ssl-socket-class" => "IO::Socket::SSL",
DEBUG:                                           "client-ssl-cipher" => "ECDHE-RSA-AES128-GCM-SHA256",
DEBUG:                                           "client-peer" => "34.230.132.157:443"
DEBUG:                                         }, 'HTTP::Headers' )
DEBUG:                  }, 'HTTP::Response' );
Here's debug info showing more details on the request and my account:

Code: Select all

DEBUG: Returning from SchedulesDirect::obtainStatus with: 
DEBUG:   $VAR1 = {
DEBUG:             "account" => {
DEBUG:                            "maxLineups" => 4,
DEBUG:                            "expires" => "2020-07-30T04:54:08Z",
DEBUG:                            "messages" => []
DEBUG:                          },
DEBUG:             "systemStatus" => [
DEBUG:                                 {
DEBUG:                                   "status" => "Online",
DEBUG:                                   "date" => "2020-04-20T10:00:00Z",
DEBUG:                                   "message" => "No known issues."
DEBUG:                                 }
DEBUG:                               ],
DEBUG:             "lastDataUpdate" => "2020-06-06T04:23:00Z",
DEBUG:             "notifications" => [],
DEBUG:             "serverID" => "20141201.web.3",
DEBUG:             "code" => 0,
DEBUG:             "datetime" => "2020-06-06T09:25:45Z",
DEBUG:             "lineups" => [
DEBUG:                            {
DEBUG:                              "name" => "Shaw Bluecurve TV - Digital",
DEBUG:                              "modified" => "2020-05-15T13:43:39Z",
DEBUG:                              "uri" => "/20141201/lineups/CAN-0006133-X",
DEBUG:                              "lineup" => "CAN-0006133-X"
DEBUG:                            }
DEBUG:                          ]
DEBUG:           };

tcordes
Posts: 4
Joined: Sat Jun 06, 2020 1:58 am

Re: initial program load fails: 500 Internal Server Error [SOLVED]

Post by tcordes »

Managed to "solve" the problem. I found where the "chunk" sizes are in tv_grab_zz_sdjson_sqlite; which determines how many items are sent in each POST request. I lowered them and now everything works!

Code: Select all

my $SD_SCHEDULE_HASH_CHUNK     = 250;          # Request stations schedules hash in chunk sizes
my $SD_SCHEDULE_CHUNK          = 200;         # Request stations schedules in chunk sizes
my $SD_PROGRAM_CHUNK           = 500;         # Request program data in chunk sizes
They were set to 250/1000/4000!! Both schedule and program needed to be drastically reduced or I kept getting 500 errors. Halving them wasn't good enough, had to go 1/5 to 1/8 the size.

I guess this could still be called a bug? Obviously the author of tv_grab_zz_sdjson_sqlite thinks his values are sane and work for him. Not sure what's different about my setup. I guess there's no harm in the smaller numbers, except that it will add more requests to the servers.

I also want to mention that for people switching from DD to SDJSON in mythtv, that as a very final step, unmentioned in the mythtv/xmltv wiki, you must run "scan for channels" in "input connections" and hit scan. Just running mythfilldatabase will not add the channels for you! I wasted a ton of time trying to put the channels in manually or in sql or whatever, but only "scan for channels" would allow recordings to actually work. I also had to write a little script to link the sqlite icon info into mythtv's channel.icon field, as nothing seems to do that for you.

gtb
Posts: 107
Joined: Thu Oct 02, 2014 2:07 pm

Re: initial program load fails: 500 Internal Server Error

Post by gtb »

A 500 error, as it says, is some internal server error in the Schedules Direct servers. While that grabber does (as it says in the output) request up to 4000 programs at a time, the API specifies that the limit is 5000, so that grabber is within the defined limits.

FWIW, I tried with your lineup (CAN-0006133-X) and I can currently download all the data without error, so the issue may have been transitory. You might want to try again.

tcordes
Posts: 4
Joined: Sat Jun 06, 2020 1:58 am

Re: initial program load fails: 500 Internal Server Error

Post by tcordes »

gtb wrote:
Sat Jun 06, 2020 10:39 pm
A 500 error, as it says, is some internal server error in the Schedules Direct servers. While that grabber does (as it says in the output) request up to 4000 programs at a time, the API specifies that the limit is 5000, so that grabber is within the defined limits.

FWIW, I tried with your lineup (CAN-0006133-X) and I can currently download all the data without error, so the issue may have been transitory. You might want to try again.
I guess I can try slowly upping the limit 100 at a time until (if) it breaks again. However, even while it generated 500 errors, it was ok with smaller chunks, so if SD will have more transient issues in the future, it might be more robust for me to just keep the lower limits? Thanks!

skst
Posts: 57
Joined: Fri Aug 17, 2007 8:13 am
Contact:

Re: initial program load fails: 500 Internal Server Error

Post by skst »

I've had this problem, too. Sometimes when you request the maximum number of programs, descriptions, schedules, etc., the SD servers choke. Usually, if you try again (and again), it'll work. I've reported the problem, but it seems to be difficult to reproduce consistently, so SD is having trouble fixing it.
USA-IL58819-X
WOW Chicago Suburbs - Digital
Naperville USA 60005

tcordes
Posts: 4
Joined: Sat Jun 06, 2020 1:58 am

Re: initial program load fails: 500 Internal Server Error

Post by tcordes »

skst wrote:
Sat Jun 06, 2020 11:29 pm
I've had this problem, too. Sometimes when you request the maximum number of programs, descriptions, schedules, etc., the SD servers choke. Usually, if you try again (and again), it'll work. I've reported the problem, but it seems to be difficult to reproduce consistently, so SD is having trouble fixing it.
It's made even harder because the sub-modules in the perl program that do the actual http requests are trying a zillion times and doing a sleep(20) after every 500. While I was working on this problem all night the other night the perl grabber would actually run for 45 mins+ and get nowhere. I did that at least 5 times. That's a lot of useless hits on the SD servers, and a lot of time wasted. If SD can't handle the bigger bunches, maybe they should lower the spec until the 500s go away. Thanks for your input!

Post Reply