Alpha version of new Schedules Direct lineup information

Discussion about Schedules Direct grabber code and data formats.
hall5714
Posts: 20
Joined: Thu Feb 07, 2013 4:34 pm

Re: Alpha version of new Schedules Direct lineup information

Post by hall5714 »

Added both WA11430 and DISH881 to my subscribed headends and the previously mentioned JSON:

{"action":"get","api":20130107,"request":["WA11430","DISH881"],"randhash":"{randhash}","object":"lineups"}

Is still only providing lineups for WA11430. To ensure it wasn't a code issue I ran the above code (with a proper randhash) at https://data2.schedulesdirect.org/request.php manually and sure enough, I get a zip file with no DISH881.json.txt in it.

EDIT: I swapped the order of the request and now DISH881 is the only found lineup. Apparently it's only downloading the first lineup it encounters.
Last edited by hall5714 on Wed Feb 20, 2013 10:45 am, edited 1 time in total.

rkulagow
SD Staff
Posts: 917
Joined: Tue Aug 14, 2007 3:15 pm

Re: Alpha version of new Schedules Direct lineup information

Post by rkulagow »

hall5714 wrote:Just realized I never thanked Robert(s) for helping my brain process how the SD API is supposed to work. Thanks to both of you!

As an aside, I noticed when running the following JSON:

{"action":"get","api":20130107,"request":["WA11430","DISH881"],"randhash":"{randhash}","object":"lineups"}

I'm getting a zip file with the WA11430.json.txt and the serverID.txt but DISH881 is no where to be found. I was doing this without any subscribed headends in my account, so I'm assuming I shouldn't be getting either (since I'm not subscribed) or both (if it doesn't matter). Thought it was curious either way.
I'll dig in and see what's happening.

hall5714
Posts: 20
Joined: Thu Feb 07, 2013 4:34 pm

Re: Alpha version of new Schedules Direct lineup information

Post by hall5714 »

I'm just working on the last bits of my python clients and had a few suggestions for the API:

(1) Send HTTP response codes when an error occurs.

I noticed if an error happens I get the appropriate HTTP response for it within the JSON (invalid username is 401, 500 for invalid json, 404 (I think) for invalid command) but the server itself is sending an HTTP 200 response. The appropriate response code should also be added as a header to the response. So if I use the wrong username/password combo I should get a HTTP 401 along with the normal JSON string. As of right now I have to verify the HTTP response was 200, parse the json input, then verify again if the response was 200 (in the json itself). If the appropriate error codes are sent along with the JSON string, I only have to check the HTTP response code (no parsing).

Besides, the whole benefit of the HTTP server is the codes... otherwise this could be implemented over a pure TCP connection :lol:

(2) Modify headend behavior so that the zipcode determines whether or not to return the subscribed headends, or all available headends.

Right now, if I pass in a randhash the server assumes I want all subscribed headends, and if I don't it assumes I want all headends available at the given zipcode. What this really should do is the following:

zip (with or without hash): all headends available in the zipcode
no zip (with hash): all subscribed headends
no zip (without hash): error

This provides consistency across the API. The only command (aside from headend) that doesn't require a randhash is getting the randhash. As of now I have to put in an escape for one command:

Code: Select all

def get_headends(self, zip=None):
    self.use_randhash = False
    # Do the call
    self.use_randhash = True
(3) Put the headend ID in the linup object and station ID in the schedules object

Currently the lineup's file name is the headend ID. However, since I'm processing everything in memory, and using a generator to return each file one by one, the headend ID is lost in translation. So that leaves me the options of (a) Put the headend ID in the lineup dict (b) Make a dict with a single entry being the headend ID, and the file being the value of the dict.

I decided to do a to avoid the overhead of creating a dictionary with one entry, so I choose (a). However, I'd much rather not mutate the data prior to returning it and if {headend:'id"} were in the lineup JSON object, I wouldn't have to). Since this is a python module, my goal is to simply take all of the data SD sends me, convert to Python dicts or lists where necessary, and give the developer an easy way to parse the data without burning up all of the memory and not having to deal with I/O overhead.

EDIT: added to (3) since stationID/schedules object is the same issue

hall5714
Posts: 20
Joined: Thu Feb 07, 2013 4:34 pm

Re: Alpha version of new Schedules Direct lineup information

Post by hall5714 »

rkulagow wrote:
hall5714 wrote:Just realized I never thanked Robert(s) for helping my brain process how the SD API is supposed to work. Thanks to both of you!

As an aside, I noticed when running the following JSON:

{"action":"get","api":20130107,"request":["WA11430","DISH881"],"randhash":"{randhash}","object":"lineups"}

I'm getting a zip file with the WA11430.json.txt and the serverID.txt but DISH881 is no where to be found. I was doing this without any subscribed headends in my account, so I'm assuming I shouldn't be getting either (since I'm not subscribed) or both (if it doesn't matter). Thought it was curious either way.
I'll dig in and see what's happening.
Thanks :). Don't think it's on my end, but I've managed to mess up easier problems so it's certainly not impossible :).

hall5714
Posts: 20
Joined: Thu Feb 07, 2013 4:34 pm

Re: Alpha version of new Schedules Direct lineup information

Post by hall5714 »

Skip #3 up there... I didn't think that one through. It's probably easier to just return the object with the generator as a method, and add a file_name method on it.

RobNewton
Posts: 4
Joined: Sun Feb 10, 2013 9:55 pm

Re: Alpha version of new Schedules Direct lineup information

Post by RobNewton »

@Hall - Are you planning to publish your work? I was going to write a python client as well.
Slugger wrote:Shameless plug :oops: , but if you want something that grabs your EPG data and produces a single zip file then might I suggest my Java grabber/API?
@Slugger - I won't be able to use Java in the environment I'm working in unfortunately. I am using your grabber for testing from CLI though. Nice work, thanks!

hall5714
Posts: 20
Joined: Thu Feb 07, 2013 4:34 pm

Re: Alpha version of new Schedules Direct lineup information

Post by hall5714 »

RobNewton wrote:@Hall - Are you planning to publish your work? I was going to write a python client as well.
https://github.com/hall5714/python-jsontv

As of right now, this thing processes all zips in memory (using generators to avoid memory problems), however it's not a TRUE client because all it does is convert the json responses into python structures (dict/lists). In the case of zipped files it opens it in memory and returns a ZippedJsonHandler object with read() function that returns a generator (so each file is handled one at a time). I haven't posted my TODO on their yet, but here's the short list:

* Backport to Python 2.7 (should be easy enough, import from __future__ with, print and do a try block on importing StringIO as BytesIO)
* It only handles server side errors, no errors in the json response. I'm hoping we will see HTTP reponse codes in responses to make this easier
* Extend ZippedJsonHandler to potentially return more than just python lists/dicts
* Add an example for SQLite
* Generate the API docs

RobNewton
Posts: 4
Joined: Sun Feb 10, 2013 9:55 pm

Re: Alpha version of new Schedules Direct lineup information

Post by RobNewton »

@Hall - thanks, that should give a huge head sart. Much appreciated. If I have any improvements during my development, I'll fork it and send you a pull request.

hall5714
Posts: 20
Joined: Thu Feb 07, 2013 4:34 pm

Re: Alpha version of new Schedules Direct lineup information

Post by hall5714 »

RobNewton wrote:@Hall - thanks, that should give a huge head sart. Much appreciated. If I have any improvements during my development, I'll fork it and send you a pull request.
Thanks Rob!

Post Reply