DataDirect replacement service - BETA 1

[rmeden]

*** TOPIC OUTDATED, see "DataDirect replacement service going live!" thread ***

I'm ready for public testing of my DataDirect replacement service! This is designed to be a drop-in replacement for the existing TMS/Gracenote DataDirect service. Apps can point to a new URL, or users can edit their local hosts file.

Thanks to all the alpha testers! I don't think any of these issues are "show stoppers" and I need better system load info. You can easily move back to TMS DD.

Known Issues:

1. Expiration Date may be slightly different. (This was caused by a bug in the SD web app, now fixed)
2. Data is only updated daily. That's the way it's going to be folks! If you want frequent updates, switch to JSON!
3. Some shows may be missing from the start of the request window. TMS DD always starts with "what's on now". The new service reports "what's on now, but only looks 4 hours before the start time. So if a show started 4 hours before the start of the request, it won't be in the data. I don't think this matters in practice (it would be in a previous day's grab), but makes diff testing a bit harder.
4. TMS DD provides additional channels. This is often a bug in TMS DD. When TMS removes a station-id from the published lineup, it is not always removed from an edited lineup. This is probably one of the causes folks need to use the "re-add lineup" option from time to time. I consider this a non-issue, SD DD is actually correct.
5. Items are in a different order. It doesn't matter to apps, but does make doing a diff harder. Sorry.
6. SD-DD Genre-Relevance only reports the most important Genre. Others will be added back later.
7. Changes to your SD lineup are not reflected on the SD-DD service. Lineup change code isn't ready yet (but close). If you need me to reset your sd-dd lineup, let me know.
8. SD-DD only appears to have 13 days of data
9. Sometimes Unicode is not always converted to UTF-8. A downloading again normally works, very strange!
10. Amazon's load balancer changes over time. If you use dd.scheduelsdirect.org this isn't a problem. If you need to use a hosts file, use this address now
    54.85.117.227 (that's a virtual IP w/o the load balancer)

The best way to use the new SD-DD service is by updating your application. In your code, replace the two TMS hostnames with "dd.schedulesdirect.org".

Code: Select all

WSDL:  http://dd.schedulesdirect.org/tech/tmsdatadirect/schedulesdirect/tvDataDelivery.wsdl
Service: http://dd.schedulesdirect.org/schedulesdirect/tvlistings/xtvdService

If you can't update your application, edit your local system hosts file to "trick" the network. (no application changes necessary!)
On Unix/Linux/Mac edit /etc/hosts (superuser required).
On Windows, edit %SYSTEMROOT%\system32\drivers\etc (normally c:\windows\..., (admin rights required, make sure your editor doesn't add ".TXT)
Add this line:

Code: Select all

54.85.117.227  docs.tms.tribune.com webservices.schedulesdirect.tmsdatadirect.com 

That's it!

I'm looking forward to feedback!

RobertE

[d0ugmac1]

Thanks for this, it certainly is significantly easier than porting a JSON implementation. Couple of things I ran into:

1. no support for 'activate' (the preferred callback time). Responds with a 400 series error I forget which. Ideally you would send a fixed 24h offset to the callin time. I have commented that piece of code for now.

2. does something different when you have multiple lineups...my app always picks the first lineup, even though it knows there are N lineups in the request. Haven't dug into this, running a single lineup for now.

3. accented characters aren't working. Probably means you have to go back and make sure you handle everything (all string processing) as UTF8 and not ANSI.

Hope that helps!

[rmeden]

#1. The function is called "acknowledge" and yes, it should work. It works for me with XMLTV's tv_grab_na_dd. If you got a 400 series error, I didn't see any post data in the HTTP request for some reason. We can troubleshoot live, or you can compare sniffer traces against tv_grab_na_dd locally and tell me what needs to be fixed.

#2. I think this could be because your application expects the returned lineups to be in the same order as TMS. There's no guarantee about that. Of course if you only need one lineup, might as well remove the other one!

#3. Accented Characters... it's been an ongoing battle.... I'm trying to keep everything UTF-8. For some strange reason I've seen cases where one call has a bad character, a second call works fine.

Robert

[gtb]

....
The easiest way to try the new SD DD service is by editing your hosts file. (no application changes necessary!)

Are there plans (on/about Nov 1st) to (request) tribune change their DNS record for webservices.schedulesdirect.tmsdatadirect.com to be a CNAME to dd.schedulesdirect.org so that existing apps continue to work "transparently"?

If so, can you please post to the forum when that change (in DNS) is complete, so that if existing apps fail (in unusual ways) people know there may be a relevant cause.

Thanks.

[rmeden]

It has been requested, but due to political sounding reasons around the breakup of Tribune it may not stick after Nov 1.

I'm thinking of asking Gracenote to add a second DNS "A" record to start sending requests to us randomly (Mid-Oct). That may cause folks to break, but it's a way to identify problems before 11/1. 10/31, I'll ask for the final DNS change to only go to us.

I suspect Tribune will shutdown the servers pretty soon after 11/1... but maybe DNS will lag and stay around longer.

I'll certainly put a clear message on the SD home page explaining to update their hosts file.

(This is of course assuming I can make the new service transparent to the old one!)

Robert

[gtb]

It has been requested, but due to political sounding reasons around the breakup of Tribune it may not stick after Nov 1.

Corporate breakups are hard to do (as far as IT is concerned). I have seen some work well, and others not so much. One can only ask. THANKS!

I'll certainly put a clear message on the SD home page explaining to update their hosts file.

RPZ (or equivalent for other DNS servers) is probably a better solution (and should perhaps be suggested for those that know what it means and have a home network that supports it), since hosts files are not always used first (various versions of my nsswitch.conf file have specified dns first), although I will agree that 99% of your customers are not going to be modifying nsswitch.conf. I admit that since I already use RPZ (for blocking purposes) adding the redirect for the tribune site becomes the obvious solution for me. If I started from scratch, I might do things differently.

(This is of course assuming I can make the new service transparent to the old one!)

I have hope. We all know that there are "abandoned"/"forgotten"/"stable" applications out there that have no chance to get updated in advance (and panic responses are never good).

Thanks for the updates.

[rmeden]

I'll certainly put a clear message on the SD home page explaining to update their hosts file.

RPZ (or equivalent for other DNS servers) is probably a better solution (and should perhaps be suggested for those that know what it means and have a home network that supports it), since hosts files are not always used first (various versions of my nsswitch.conf file have specified dns first), although I will agree that 99% of your customers are not going to be modifying nsswitch.conf. I admit that since I already use RPZ (for blocking purposes) adding the redirect for the tribune site becomes the obvious solution for me. If I started from scratch, I might do things differently.

I disagree.. anyone who uses RPZ would know to use it and not edit /etc/hosts. The instructions are for the less technical, the rest can translate.

[rmeden]

I updated the first post, but I'll mention it here as well.

The SD-DD service now only returns one Genre (with a relevance of 0). Multiple Genres starting with 1 in random order was causing problems for MythTV.

The missing Genres will be added later.

[rmeden]

Current IP ( *.205) updated in the Original Post.

[wgmaker]

Our SD client based on this DataDirect replacement service is ready. Is it OK to publish it ? If not could you give a date?
regards Jan (WebGrab+Plus client)