DataDirect replacement service going live!

Forum rules
Get Help using the Schedules Direct service. NOTE: application issues are probably better resolved via the application's support methods. If you post here, at least include your application name!

Re: DataDirect replacement service going live!

Postby sgraves » Sun Nov 09, 2014 5:04 pm

Nevermind, I think I know what the issue is.

Since you're now in control of the service, can you provide updated the WSDL and XSD anyways? I plan to switch to JSON anyways, but would be nice.
sgraves
 
Posts: 5
Joined: Sun Nov 09, 2014 12:54 pm

Re: Tribune sending requests to new SD-DD service

Postby rmeden » Sun Nov 09, 2014 5:39 pm

sgraves wrote:I'm unable to reach the new server. It continues to time-out trying to connect. Is it currently active?


If you get timeouts connecting to the new server and the SD web page doesn't report any errors, you may have been blocked due to excessive attempts.

There is at least one application that immediately retries after an erorr (even account expired!) causing thousands of hits against the server. I've configured fail2ban to block what I consider abuse. (right now 40 requests an hour I think).

That could cause problems doing development. If you see that, contact admin@sd and I can whitelist you.

I'm sorry to have to do it, but some people were hitting tribune (and us!) us over 10k times a day!

sgraves wrote:Since you're now in control of the service, can you provide updated the WSDL and XSD anyways? I plan to switch to JSON anyways, but would be nice.


The WSDL is available. http://dd.schedulesdirect.org/tech/tmsd ... ivery.wsdl

Robert
rmeden
SD Board Member
 
Posts: 1486
Joined: Tue Aug 14, 2007 2:31 pm
Location: Cedar Hill, TX

Re: Tribune sending requests to new SD-DD service

Postby sgraves » Sun Nov 09, 2014 6:50 pm

rmeden wrote:
sgraves wrote:I'm unable to reach the new server. It continues to time-out trying to connect. Is it currently active?


If you get timeouts connecting to the new server and the SD web page doesn't report any errors, you may have been blocked due to excessive attempts.

There is at least one application that immediately retries after an erorr (even account expired!) causing thousands of hits against the server. I've configured fail2ban to block what I consider abuse. (right now 40 requests an hour I think).

That could cause problems doing development. If you see that, contact admin@sd and I can whitelist you.

Robert


Oops.... I just checked my app's server logs, and I'm retrying immediately on failure. You're probably seeing a retry ~20 seconds. I'll fix the code tonight - never implemented max retry per hour. It's worked for years without issue, so I never revisited it. Yeah, definitely banned. I'll email you once the fix in place.
sgraves
 
Posts: 5
Joined: Sun Nov 09, 2014 12:54 pm

Re: DataDirect replacement service going live!

Postby rmeden » Sun Nov 09, 2014 8:56 pm

best solution is an exponential back-off based on the error. Of course no reason to retry a login failure or expired account... but other errors may be worth a retry. 33sec, 66sec, 132sec, etc. Stop at a reasonable point.
rmeden
SD Board Member
 
Posts: 1486
Joined: Tue Aug 14, 2007 2:31 pm
Location: Cedar Hill, TX

Re: DataDirect replacement service going live!

Postby sgraves » Sun Nov 09, 2014 10:05 pm

Thanks and sorry again. I retry every 15 minutes on failure, but will bump this up to 1 hour after further testing. I only update once per day when successful. A bug was causing an infinite retry.

Seems like the dolby attribute is coming in with a value of "DD5.1" which isn't defined in the 'tmsxtvdv1.xsd':
Code: Select all
<xsd:attribute name="dolby" use="optional">
  <xsd:annotation>
     <xsd:documentation xml:lang="en">Indicates whether the program audio is broadcast in Dolby, or Dolby Digital.</xsd:documentation>
  </xsd:annotation>
  <xsd:simpleType>
     <xsd:restriction base="xsd:string">
        <xsd:enumeration value="Dolby"/>
        <xsd:enumeration value="Dolby Digital"/>
     </xsd:restriction>
 </xsd:simpleType>
</xsd:attribute>


Example result:
Code: Select all
<schedule program="EP017839560069" station="19616" time="2014-11-09T05:00:00Z" duration="PT01H00M" tvRating="TV-14" stereo="true" hdtv="true" closeCaptioned="true" dolby="DD5.1" />


Easy to workaround, but was wondering if there's a new XSD in the works.
sgraves
 
Posts: 5
Joined: Sun Nov 09, 2014 12:54 pm

Re: DataDirect replacement service going live!

Postby rmeden » Sun Nov 09, 2014 11:49 pm

sgraves wrote:Seems like the dolby attribute is coming in with a value of "DD5.1" which isn't defined in the 'tmsxtvdv1.xsd'


Fixed.. I'll roll it out tomorrow (no code roll-outs at 1am!)

Robert
rmeden
SD Board Member
 
Posts: 1486
Joined: Tue Aug 14, 2007 2:31 pm
Location: Cedar Hill, TX

MythTV: lineup map glitch after switching

Postby cejaekl » Tue Nov 11, 2014 8:38 pm

(First time poster, so please be gentle if I'm in the wrong spot.)

I noticed yesterday that my MythTV guide data was only present for some channels, but not for others. I use an OTA antenna to receive ATSC in Ottawa, Canada. To give examples, channels 4 (CBC-English) and 9 (CBC-French) had guide data, but channel 13 (CTV-One) did not. Things were working fine a couple of weeks ago, so I'm guessing that this problem had something to do with the transition away from the Tribune servers.

Looking at the XML being sent by schedules direct, I saw this snippet (postal code XXX'd out--I'm mildly paranoid):
Code: Select all
<lineup id='PC:XXXX' name='Local Broadcast Listings' location='Antenna' type='LocalBroadcast' postalCode='XXXXXX'>
<map station='10105' channel='60'/>
<map station='10112' channel='11'/>
<map station='10116' channel='43'/>
<map station='10119' channel='24'/>
<map station='10125' channel='65'/>
<map station='10127' channel='13'/>
<map station='15984' channel='40'/>
<map station='15985' channel='30'/>
<map station='16160' channel='6'/>
<map station='18805' channel='32'/>
<map station='31558' channel='14'/>
<map station='52235' channel='9' channelMinor='1'/>
<map station='58494' channel='4' channelMinor='1'/>
<map station='64990' channel='27' channelMinor='1'/>
<map station='64990' channel='60' channelMinor='1'/>
<map station='64992' channel='20' channelMinor='1'/>
<map station='64994' channel='14' channelMinor='1'/>
<map station='64994' channel='66' channelMinor='1'/>
<map station='68025' channel='13' channelMinor='1'/>
<map station='68285' channel='17' channelMinor='1'/>
<map station='68285' channel='65' channelMinor='1'/>
<map station='72715' channel='43' channelMinor='1'/>
<map station='73010' channel='6' channelMinor='1'/>
<map station='73016' channel='24' channelMinor='1'/>
<map station='73107' channel='34' channelMinor='1'/>
<map station='73190' channel='42' channelMinor='1'/>
<map station='73192' channel='11' channelMinor='1'/>
<map station='73196' channel='30' channelMinor='1'/>
<map station='73198' channel='40' channelMinor='1'/>
</lineup>


So... some channels have two station identifier codes, a lower-numbered one that doesn't include a subchannel (channelMinor), and a higher-numbered one that does. Scanning further down the XML data, it seemed that the program listings that were present all referred back to the higher-numbered identifier. Comparing with the channel ids stored in my MythTV database's channel table, I could see that the channels for which I had no guide data were using the lower-numbered station identifier. After updating that table in the database to contain the higher-numbered identifier and re-running mythfilldatabase to fetch the guide data again, I now have listings for all of my channels.

Speculating here (I don't have records to confirm this), I suspect that the older identifier was the pre-digital-transition analog channel. I seem to recall that, at least for a while after the digital switchover, SchedulesDirect had two listings for some of our local channels, one analog (e.g. CHCH-11) and another digital (e.g. CHCH-11DT). I'm guessing that the old analog station identifier continued to work, but that the new compatibility layer is only pushing out guide data for the new digital channel identifier.

I don't know how many people this affects, but figured I'd post something in the hope that it'll help someone else figure out how to fix their setup.
cejaekl
 
Posts: 1
Joined: Tue Nov 11, 2014 7:25 pm

Re: DataDirect replacement service going live!

Postby rmeden » Tue Nov 11, 2014 10:05 pm

That's pretty strange... all modern stuff should be ATSC with major/minor channel ids. It sounds like that part of the lookup isn't working.

1. Try removing your lineup and adding it back.
2. If that doesn't fix it, email admin@sd with your postal code or open a ticket... I'll check the DB.

Robert
rmeden
SD Board Member
 
Posts: 1486
Joined: Tue Aug 14, 2007 2:31 pm
Location: Cedar Hill, TX

Re: DataDirect replacement service going live!

Postby jsmar » Wed Nov 12, 2014 3:13 am

So I thought after Nov 1 you were going to enable digest authentication, so I just put in the hard coded TMS IP (144.142.232.53) until then (the 50% didn't appear to work for me, I always seemed to get the sd server and therefore an authentication failure).

Anyway, although the hard coded IP appears to still be "working" I don't think the data is being updated. I tried switching to sd again, but digest authentication isn't enabled yet it would appear. Will that be happening soon?

I can't support https and I'd rather not send my account information unencrypted. Plus I'd have to write the support for basic authentication. Am I the only one left who wants digest authentication to work?

John
jsmar
 
Posts: 3
Joined: Thu Jun 18, 2009 12:06 am

Re: DataDirect replacement service going live!

Postby rmeden » Wed Nov 12, 2014 9:14 am

Digest authentication has been enabled for a while... Basic is still the requested one though.

We may need to troubleshoot the problem. Contact Admin@SD no reason to do it in the forum.
rmeden
SD Board Member
 
Posts: 1486
Joined: Tue Aug 14, 2007 2:31 pm
Location: Cedar Hill, TX

PreviousNext

Return to Support

Who is online

Users browsing this forum: No registered users and 6 guests

cron