Need help figureing out where my problem is.

Forum rules
Get Help using the Schedules Direct service. NOTE: application issues are probably better resolved via the application's support methods. If you post here, at least include your application name!

Need help figureing out where my problem is.

Postby AndrewHogan » Mon Nov 10, 2014 9:21 am

Last month, partly to keep my system current with SD moving to JSON, and partly because I was unhappy with my cable provider, I built a new MythTV backend (Mythbuntu) with an OTA tuner card. I made sure it was updated to Mythtv 0.27.4 which I'd read was supposed to be able to use the JSON side of things.

In early Nov (I think it was the 2nd or 3rd) I ran a mythfilldatabase to see if it was still working and I received new data, I thought everything was good.

Last night a show I was expecting to watch was missing, and when I went into the listings I found a 3.5 day gap in my data. I have no data from 6am Sunday 2014-11-09 to 6pm Wed 2014-11-12. There also seems to be a gap in next week's data.

At first I assumed this was a problem with Mythfilldatabase, so I tried several versions of running in manually. Then I wanted to try and make it work off a file so I could see the data it was working from. I followed the instructions here ... http://forums.schedulesdirect.org/viewtopic.php?f=6&t=295 to try and get a file, but I could not get past the step
Code: Select all
tv_grab_na_dd --configure --dd-data out.xml

I would get this result
Code: Select all
Fetching from Schedules Direct     Fetched 0 k/bytes in 0 seconds
loading data:
WARNING: error parsing DD xml:
not well-formed (invalid token) at line 1, column 3, byte 3 at /usr/lib/perl5/XML/Parser.pm line 187.
 at /usr/bin/tv_grab_na_dd line 928.


ERROR: Bad XML from DD, cannot continue. Consider using --dropbadchar or Capture xml with --dd-data

and the out.xml file contained
Code: Select all
Can't connect to webservices.schedulesdirect.tmsdatadirect.com:80 (Connection refused)

LWP::Protocol::http::Socket: connect: Connection refused at /usr/share/perl5/LWP/Protocol/http.pm line 41, <STDIN> line 3.

I can ping webservices.schedulesdirect.tmsdatadirect.com so it's not a routing issue, but I can't connect to port 80 on it either from my house, or from my phone with a different public IP off the mobile network(in case all my testing of mythfilldatabase got my IP blacklisted)

Should tv_grab_na_dd have worked? Is that now outdated and I need to find some other way to receive complete (now JSON formatted) file, to manually feed to mythfilldatabase for testing? http://www.schedulesdirect.org/getdata works, but it's only a partial set of data, and it's still XML formatted.

I don't know where to go from here. Any help would be greatly appreciated.

Thanks in advance.
AndrewHogan
 
Posts: 4
Joined: Mon Jan 11, 2010 11:15 pm

Re: Need help figureing out where my problem is.

Postby rkulagow » Mon Nov 10, 2014 9:58 am

The JSON service is entirely different than the XML service; different grabber, different configuration method.

If you're running mythfilldatabase, you're still on the XML service.
rkulagow
SD Staff
 
Posts: 911
Joined: Tue Aug 14, 2007 3:15 pm

Re: Need help figureing out where my problem is.

Postby rmeden » Mon Nov 10, 2014 10:11 am

You're using the SD-DD service, not JSON.

You were banned for 24 hours after 54 downloads in an hour yesterday at 19:40 EST. It would have reset today. (also use --dd-grab-all )

Check your cron entry, there's no point in getting data more than once.

Robert
rmeden
SD Board Member
 
Posts: 1511
Joined: Tue Aug 14, 2007 2:31 pm
Location: Cedar Hill, TX

Re: Need help figureing out where my problem is.

Postby bill6502 » Mon Nov 10, 2014 10:40 am

rmeden wrote:...
You were banned for 24 hours after 54 downloads in an hour yesterday at 19:40 EST. It would have reset today. (also use --dd-grab-all )

Check your cron entry, there's no point in getting data more than once.

Hi Robert,

--dd-grab-all generates 3 HTTP requests (Authentication, Download, Acknowledge.) So if a user ran it 14
times in one hour (for testing, not for production), they'd get banned. It that correct? Or is the ban based
on the number of files downloaded (in which case it would be 20 (Guide download + Acknowledge?)

Just a suggestion about cron. Users can run mythtv-setup to let the backend start mythfilldatabase. Then
it will be run at the time suggested by SD. If run by cron, the suggested time isn't used.
Code: Select all
# Automatically update program listings: Checked
# Guide data program: mythfilldatabase
# Guide data arguments: --dd-grab-all
# Guide data program execution start: 0
# Guide data program execution end: 23
# Run guide data program at time suggested by the grabber: Checked
bill6502
 
Posts: 16
Joined: Fri Sep 19, 2014 4:52 pm

Re: Need help figureing out where my problem is.

Postby AndrewHogan » Mon Nov 10, 2014 12:38 pm

Around 19:40 EST yesterday I was running mythfilldatabase a bunch of times manually to try and fix the gaps in my listings.

--dd-grab-all was the third option I tried, and it didn't fill the gaps in my information. (After mythfilldatabase with no parameters, and then with --refresh)

I then tried several variations on --refresh and --refresh-all. None of which would fill the gaps.

So I guess I was blacklisted before I attempted to get the file with tv_grab_na_dd

Running tv_grab_na_dd this morning before I posted here gave me the same errors as last night, but just now mythfilldatabase with --dd-grab-all filled the gaps, and tv_grab_na_dd worked.


Ugh I still have no idea why there were gaps, or why they went away, but at least for now my listings look the way they should.
Also I was assuming based off comments about 0.27.4 being ready for the change to SD that they were adding the ability to use the JSON feed to mythfilldatabase. Clearly that was a bad assumption. It sounds like I should stick with mythfilldatabase anyway though. Does that sound like the right choice to y'all? (I might not be a southerner, but the addition of a plural second person pronoun really makes sense to me)

Thanks for all the information and suggestions.
AndrewHogan
 
Posts: 4
Joined: Mon Jan 11, 2010 11:15 pm

Re: Need help figureing out where my problem is.

Postby rmeden » Mon Nov 10, 2014 2:07 pm

bill6502 wrote:
rmeden wrote:...
You were banned for 24 hours after 54 downloads in an hour yesterday at 19:40 EST. It would have reset today. (also use --dd-grab-all )

Check your cron entry, there's no point in getting data more than once.

Hi Robert,

--dd-grab-all generates 3 HTTP requests (Authentication, Download, Acknowledge.) So if a user ran it 14
times in one hour (for testing, not for production), they'd get banned. It that correct? Or is the ban based
on the number of files downloaded (in which case it would be 20 (Guide download + Acknowledge?)

Yea, That sounds right. But I still think 14 tests in an hour is quite a bit. I've been doing a lot of testing and only hit it once myself. It's amazing how many bad hits we get. I plan on adding a notice to the SD web page if you connect from a banned IP.

I also plan on tweaking the algorithm based on response types. 402 (account expired) will get you banned faster than a 401 (bad password, password request). I may also allow more 200 (success) calls.

Robert
rmeden
SD Board Member
 
Posts: 1511
Joined: Tue Aug 14, 2007 2:31 pm
Location: Cedar Hill, TX


Return to Support

Who is online

Users browsing this forum: No registered users and 9 guests

cron