Developer Wishlish for new JSON API

Developer Wishlish for new JSON API

Postby rkulagow » Fri Feb 22, 2013 9:12 am

Here's your chance to get input into what you'd like to see / what would make your life simpler. Can't guarantee that everything is going to be possible, especially since this is a side project.
rkulagow
SD Staff
 
Posts: 915
Joined: Tue Aug 14, 2007 3:15 pm

Re: Developer Wishlish for new JSON API

Postby hall5714 » Fri Feb 22, 2013 11:25 am

Posted this on the other thread, so I'll summarize here:

(1) Send HTTP response codes when an error occurs.
IE: Incorrect password should send an HTTP 401, etc. Since the JSON response is already including HTTP codes, this should be a simple matter of header('X-Response-Code: 404', true, 404); (or header_response_code(404) if you are running PHP 5.4+).

(2) Modify headend call behavior so that the zipcode determines whether or not to return the subscribed headends, or all available headends.
As of now my query builder as to "turn off" sending the randhash if we want data from a zipcode. If a zipcode is included, return all headends at that zipcode, if not return only the subscribed headends. alternatively have separate commands (subscribed_headends and available_headends).

(3) Remove the "submit=submit" requirement from the post object.
If our POST line doesn't have "submit=submit" it just brings up the page form. It shouldn't be expecting a form submission, a simple request=POST_DATA should be enough.

(4) Sending zips should be consistent
If our command is successful, it throws a zip file at as, if it's not, it throws a json response string at us. Instead of sending a zip, it should send a json string with the "data" object being the zip bytes, or a temp "link" to grab the data. The temp link is nice because it only has to be active ~15 minutes, so if the client has some sort of error during transmission, we can download it again without the server having to rebuild the zip file (although you technically may be doing that server-side anyway).

If I were to order by preference (1) would be HUGE... it would greatly simplify error checking for clients. (2) would be a close second as it alleviates having to "remove" the hashcode from the query just to get available headends. 3 and 4 are more "it would be neat" but it's not going to ruin anyones day if they don't get implemented. :)
hall5714
 
Posts: 20
Joined: Thu Feb 07, 2013 4:34 pm

Re: Developer Wishlish for new JSON API

Postby rkulagow » Sun Feb 24, 2013 9:11 am

(1) Send HTTP response codes when an error occurs.
IE: Incorrect password should send an HTTP 401, etc. Since the JSON response is already including HTTP codes, this should be a simple matter of header('X-Response-Code: 404', true, 404); (or header_response_code(404) if you are running PHP 5.4+).

- earlier versions of the code did this. It would be relatively simple to put it back. If the other active developers can weigh in, I can make the change. The thought was that I wanted to differentiate between response codes from the API and apache response codes; if apache dies for whatever reason, you'd get a 404 or something, and the application could differentiate between a 404 from the server versus the API. A 200 from apache could still result in a 404 from the API. Input?

(2) Modify headend call behavior so that the zipcode determines whether or not to return the subscribed headends, or all available headends.
As of now my query builder as to "turn off" sending the randhash if we want data from a zipcode. If a zipcode is included, return all headends at that zipcode, if not return only the subscribed headends. alternatively have separate commands (subscribed_headends and available_headends).

- This was originally implemented so that users that are using the QAM scanner application I wrote could scan without having a Schedules Direct account. I've come to the conclusion that anyone who wants the QAM data would need an account anyway. The "get" "headends" will require a randhash moving forward. I think the following scheme should work:

"get" "headends" "randhash" "request":"PC:60030" (or whatever) will return all headends for the postal code. Randhash is required.
"get" "headends" "randhash" "request" "subscribed" - returns the headends that the user has added to their account. Randhash is required.

(3) Remove the "submit=submit" requirement from the post object.
If our POST line doesn't have "submit=submit" it just brings up the page form. It shouldn't be expecting a form submission, a simple request=POST_DATA should be enough.

- Should be pretty simple. Any other active developers have an issue with this? The original submit=submit was back when the API was being developed with a web browser. Now that everything is through API it can probably be removed.

(4) Sending zips should be consistent
If our command is successful, it throws a zip file at as, if it's not, it throws a json response string at us. Instead of sending a zip, it should send a json string with the "data" object being the zip bytes, or a temp "link" to grab the data. The temp link is nice because it only has to be active ~15 minutes, so if the client has some sort of error during transmission, we can download it again without the server having to rebuild the zip file (although you technically may be doing that server-side anyway).

- Interesting. Let me think about that some more.

If I were to order by preference (1) would be HUGE... it would greatly simplify error checking for clients. (2) would be a close second as it alleviates having to "remove" the hashcode from the query just to get available headends. 3 and 4 are more "it would be neat" but it's not going to ruin anyones day if they don't get implemented. :)
rkulagow
SD Staff
 
Posts: 915
Joined: Tue Aug 14, 2007 3:15 pm

Re: Developer Wishlish for new JSON API

Postby hall5714 » Sun Feb 24, 2013 9:49 am

rkulagow wrote:- earlier versions of the code did this. It would be relatively simple to put it back. If the other active developers can weigh in, I can make the change. The thought was that I wanted to differentiate between response codes from the API and apache response codes; if apache dies for whatever reason, you'd get a 404 or something, and the application could differentiate between a 404 from the server versus the API. A 200 from apache could still result in a 404 from the API. Input?


There's two ways to handle that I suppose. First is server-side, by sending all response codes with an added "X-JSON-Error=True" (or something of that nature) so users can peek at the headers if they want to know if it's JSON or Apache causing the problem. However my way of resolving that is simply trying to parse the content:

Code: Select all
        try: # only works with valid JSON
            self.is_json = True
            return json.loads(self.response.text)
        except TypeError:
            self.is_json = False

self.response.raise_for_status()
# if self.is_json is true and an HTTPError was raised, it's a json error
# if self.is_json is false and an HTTPError was raised, it's an apache error


I would have to change to content-type header checking for zip files, but I can live with that (depending on how my option 4 works out).

This was originally implemented so that users that are using the QAM scanner application I wrote could scan without having a Schedules Direct account. I've come to the conclusion that anyone who wants the QAM data would need an account anyway. The "get" "headends" will require a randhash moving forward. I think the following scheme should work:

"get" "headends" "randhash" "request":"PC:60030" (or whatever) will return all headends for the postal code. Randhash is required.
"get" "headends" "randhash" "request" "subscribed" - returns the headends that the user has added to their account. Randhash is required.


Perfect :)

Should be pretty simple. Any other active developers have an issue with this? The original submit=submit was back when the API was being developed with a web browser. Now that everything is through API it can probably be removed.


This one isn't "that" big of a deal... just think it cleans up my request object a little more.

- Interesting. Let me think about that some more.


Cool. And I suppose this would actually make my first suggestion leagues easier as well: the server should ALWAYS return json... if there's a json decode error, then we know it's Apache sending the error code.

Thanks for the response, and if you need help with anything please let me know!
hall5714
 
Posts: 20
Joined: Thu Feb 07, 2013 4:34 pm

Re: Developer Wishlish for new JSON API

Postby rkulagow » Sun Feb 24, 2013 2:11 pm

(2) Modify headend call behavior so that the zipcode determines whether or not to return the subscribed headends, or all available headends.
As of now my query builder as to "turn off" sending the randhash if we want data from a zipcode. If a zipcode is included, return all headends at that zipcode, if not return only the subscribed headends. alternatively have separate commands (subscribed_headends and available_headends).

- This was originally implemented so that users that are using the QAM scanner application I wrote could scan without having a Schedules Direct account. I've come to the conclusion that anyone who wants the QAM data would need an account anyway. The "get" "headends" will require a randhash moving forward. I think the following scheme should work:

"get" "headends" "randhash" "request":"PC:60030" (or whatever) will return all headends for the postal code. Randhash is required.
"get" "headends" "randhash" "request" "subscribed" - returns the headends that the user has added to their account. Randhash is required.


This has been implemented on the beta server.

API has been bumped to 20130224.

Give it a whirl and let me know.
rkulagow
SD Staff
 
Posts: 915
Joined: Tue Aug 14, 2007 3:15 pm

Re: Developer Wishlish for new JSON API

Postby Slugger » Sun Feb 24, 2013 9:12 pm

hall5714 wrote:Posted this on the other thread, so I'll summarize here:

(1) Send HTTP response codes when an error occurs.
IE: Incorrect password should send an HTTP 401, etc. Since the JSON response is already including HTTP codes, this should be a simple matter of header('X-Response-Code: 404', true, 404); (or header_response_code(404) if you are running PHP 5.4+).


Robert's explanation was perfect. Only because the error codes the API generates appear to be the same as HTTP status codes, but in reality they aren't. The HTTP status code should reflect the status of the web server itself and when it's behaving as expected, it should only ever return 2xx status codes. In other words, you should always get a "successful" response from the web server. What does that response mean in the context of the JSON API? Well you need to parse the JSON and look at its status code.

If you wanted to add an HTTP header like "X-API-Status" or something and add that to the HTTP response then fine, but you're probably not gaining much. But I definitely would continue to differentiate HTTP status codes from API status codes. If auth for the server was handled via HTTP basic auth then throwing back 401 HTTP statuses would make sense, but since that isn't the case then the web server should never throw a 401 or 403 error.

(2) Modify headend call behavior so that the zipcode determines whether or not to return the subscribed headends, or all available headends.
As of now my query builder as to "turn off" sending the randhash if we want data from a zipcode. If a zipcode is included, return all headends at that zipcode, if not return only the subscribed headends. alternatively have separate commands (subscribed_headends and available_headends).


I think I see this has already changed on the test server. I'll comment on that later. :)

(3) Remove the "submit=submit" requirement from the post object.
If our POST line doesn't have "submit=submit" it just brings up the page form. It shouldn't be expecting a form submission, a simple request=POST_DATA should be enough.


This makes sense, but I assume you'll just ignore this field and if an older client continues to send it, you won't care?

(4) Sending zips should be consistent
If our command is successful, it throws a zip file at as, if it's not, it throws a json response string at us. Instead of sending a zip, it should send a json string with the "data" object being the zip bytes, or a temp "link" to grab the data. The temp link is nice because it only has to be active ~15 minutes, so if the client has some sort of error during transmission, we can download it again without the server having to rebuild the zip file (although you technically may be doing that server-side anyway).


I thought about suggesting this a long time ago, but did you know that, by definition, JSON only supports unicode characters in its payload? So to pass zipped/compressed data in json means you'd have to encode it, probably base64 encode. Not the end of the world, but I don't think a lot of json libs are going to handle 15, 20, up to 30MB json strings all too well. Though I'm certainly not against experimenting, but my gut tells me it would be a minor disaster for some (many? all?) json libs; maybe even php's json lib server side.

However, I really like the idea, it really normalizes the api response (i.e. each and every request returns a json struct in response). That's why I was going to suggest it to Robert a long time ago, but balked at the idea when determining the only way that was going to happen was if the zipped data was base64 encoded. Robert and I ran into problems with the php server when trying to get it to feed back massive json objects in earlier alpha iterations of the api.

If I were to order by preference (1) would be HUGE... it would greatly simplify error checking for clients. (2) would be a close second as it alleviates having to "remove" the hashcode from the query just to get available headends. 3 and 4 are more "it would be neat" but it's not going to ruin anyones day if they don't get implemented. :)


#1 shouldn't be changed, imho. I'm indifferent to the rest of the ideas.
Slugger
 
Posts: 77
Joined: Sun Sep 18, 2011 1:22 pm

Re: Developer Wishlish for new JSON API

Postby Slugger » Sun Feb 24, 2013 9:16 pm

rkulagow wrote:
(2) Modify headend call behavior so that the zipcode determines whether or not to return the subscribed headends, or all available headends.
As of now my query builder as to "turn off" sending the randhash if we want data from a zipcode. If a zipcode is included, return all headends at that zipcode, if not return only the subscribed headends. alternatively have separate commands (subscribed_headends and available_headends).

- This was originally implemented so that users that are using the QAM scanner application I wrote could scan without having a Schedules Direct account. I've come to the conclusion that anyone who wants the QAM data would need an account anyway. The "get" "headends" will require a randhash moving forward. I think the following scheme should work:

"get" "headends" "randhash" "request":"PC:60030" (or whatever) will return all headends for the postal code. Randhash is required.
"get" "headends" "randhash" "request" "subscribed" - returns the headends that the user has added to their account. Randhash is required.


This has been implemented on the beta server.

API has been bumped to 20130224.

Give it a whirl and let me know.


Idea is fine, one suggestion on implementation. :)

Instead of "request":"subscribed", why not make it "request":null? JSON has a null type and it seems to be perfect for this scenario. If the request is null then return the user's subscribed headends. If the request is not null then it's a zip code so return all headends for it. Magic strings are evil, especially in a public API definition, imho. I think you can avoid the magic string "subscribed" in this case.
Slugger
 
Posts: 77
Joined: Sun Sep 18, 2011 1:22 pm

Re: Developer Wishlish for new JSON API

Postby rkulagow » Sun Feb 24, 2013 10:29 pm

"get" "headends" "randhash" "request":"PC:60030" (or whatever) will return all headends for the postal code. Randhash is required.
"get" "headends" "randhash" "request" "subscribed" - returns the headends that the user has added to their account. Randhash is required.

This has been implemented on the beta server.

API has been bumped to 20130224.

Give it a whirl and let me know.


Idea is fine, one suggestion on implementation. :)

Instead of "request":"subscribed", why not make it "request":null? JSON has a null type and it seems to be perfect for this scenario. If the request is null then return the user's subscribed headends. If the request is not null then it's a zip code so return all headends for it. Magic strings are evil, especially in a public API definition, imho. I think you can avoid the magic string "subscribed" in this case.


OK, please try it again, and send an empty request.

I don't like sending the .zip data within the response for exactly the reason that slugger already stated; it doesn't scale with the json encoder in PHP. I will look at putting the .zip generated files onto S3 and providing a link in the response. That will get my "little" servers out of the business of serving files; let Amazon worry about it. :)
rkulagow
SD Staff
 
Posts: 915
Joined: Tue Aug 14, 2007 3:15 pm

Re: Developer Wishlish for new JSON API

Postby hall5714 » Mon Feb 25, 2013 12:01 am

Slugger wrote:Robert's explanation was perfect. Only because the error codes the API generates appear to be the same as HTTP status codes, but in reality they aren't. The HTTP status code should reflect the status of the web server itself and when it's behaving as expected, it should only ever return 2xx status codes. In other words, you should always get a "successful" response from the web server. What does that response mean in the context of the JSON API? Well you need to parse the JSON and look at its status code.

If you wanted to add an HTTP header like "X-API-Status" or something and add that to the HTTP response then fine, but you're probably not gaining much. But I definitely would continue to differentiate HTTP status codes from API status codes. If auth for the server was handled via HTTP basic auth then throwing back 401 HTTP statuses would make sense, but since that isn't the case then the web server should never throw a 401 or 403 error.


Responding to resource requests with appropriate HTTP response codes is the entire foundation of REST:

https://dev.twitter.com/docs/error-codes-responses
http://developer.netflix.com/docs/HTTP_Status_Codes
http://developer.yahoo.com/social/rest_ ... codes.html

HTTP response codes are status codes in reference to a resource, whether that resource is a web page or a Schedules Direct program is entirely moot. HTTP response are a means to notify the client that there was a problem with a requested resource, whether that resource is myCoolPage.php or a programID that doesn't exist. They aren't "status codes" they are "response status codes", they are a numeric response to a request telling the client what the status of that request is. So if I get a HTTP 200 OK when I requested a non-existing resource, then we are breaking the protocol :D. If Apache raises a 404 error, it is because there was a problem locating a resource that was requested. If my client requests a programID that doesn't exist, then SD should raise a 404 error, because it had a problem locating a resource that I requested.

I guess my "short" response would be, if we aren't going to use HTTP Status codes, why use HTTP at all? Why not JSON over TCP Sockets?

</end_rant> :lol:

I thought about suggesting this a long time ago, but did you know that, by definition, JSON only supports unicode characters in its payload? So to pass zipped/compressed data in json means you'd have to encode it, probably base64 encode. Not the end of the world, but I don't think a lot of json libs are going to handle 15, 20, up to 30MB json strings all too well. Though I'm certainly not against experimenting, but my gut tells me it would be a minor disaster for some (many? all?) json libs; maybe even php's json lib server side.


Yeah I wasn't aware of it, but thought it was a possibility. I've only really played with JSON in javascript and Python so I've got limited experience... and Python has an impressive json library and well, javascript = JSON :P.
hall5714
 
Posts: 20
Joined: Thu Feb 07, 2013 4:34 pm

Re: Developer Wishlish for new JSON API

Postby Slugger » Mon Feb 25, 2013 5:47 am

hall5714 wrote:
Slugger wrote:Robert's explanation was perfect. Only because the error codes the API generates appear to be the same as HTTP status codes, but in reality they aren't. The HTTP status code should reflect the status of the web server itself and when it's behaving as expected, it should only ever return 2xx status codes. In other words, you should always get a "successful" response from the web server. What does that response mean in the context of the JSON API? Well you need to parse the JSON and look at its status code.

If you wanted to add an HTTP header like "X-API-Status" or something and add that to the HTTP response then fine, but you're probably not gaining much. But I definitely would continue to differentiate HTTP status codes from API status codes. If auth for the server was handled via HTTP basic auth then throwing back 401 HTTP statuses would make sense, but since that isn't the case then the web server should never throw a 401 or 403 error.


Responding to resource requests with appropriate HTTP response codes is the entire foundation of REST:

https://dev.twitter.com/docs/error-codes-responses
http://developer.netflix.com/docs/HTTP_Status_Codes
http://developer.yahoo.com/social/rest_ ... codes.html

HTTP response codes are status codes in reference to a resource, whether that resource is a web page or a Schedules Direct program is entirely moot. HTTP response are a means to notify the client that there was a problem with a requested resource, whether that resource is myCoolPage.php or a programID that doesn't exist. They aren't "status codes" they are "response status codes", they are a numeric response to a request telling the client what the status of that request is. So if I get a HTTP 200 OK when I requested a non-existing resource, then we are breaking the protocol :D. If Apache raises a 404 error, it is because there was a problem locating a resource that was requested. If my client requests a programID that doesn't exist, then SD should raise a 404 error, because it had a problem locating a resource that I requested.

I guess my "short" response would be, if we aren't going to use HTTP Status codes, why use HTTP at all? Why not JSON over TCP Sockets?

</end_rant> :lol:


I don't think much of any of this looks like a RESTful service. If it did, I'd be inclined to agree with you. However, this whole service works on sending requests to request.php and getting back responses. And request.php (the lone resource) always exists so unless the Apache server is ill, Apache should always return status 200. Now if the requests were changed to be RESTful then it would make sense to have the http status code reflect the status of the reqeusted resource.

With that said, it's, of course, Robert's call. Making this change will break the Java client.
Slugger
 
Posts: 77
Joined: Sun Sep 18, 2011 1:22 pm

Next

Return to Developer

Who is online

Users browsing this forum: No registered users and 3 guests

cron