Page 1 of 4

Problems connecting to sd-dd servers ( blocking )

Posted: Thu Nov 13, 2014 5:49 pm
by rmeden
One problem we've inherited from Tribune taking over the DataDirect service is poorly written applications abusing the service. (This only concerns the SD-DD service, not JSON)

Data is updated once a day. There really isn't any benefit in checking for data more than that (per device). If your account is expired, attempting to download data 10,000 times a day (as two people are doing) is downright annoying. Many with expired accounts have automated processes attempting thousands of times a day.

To combat this, I've implemented some filtering. Once you've been blocked, the data server will refuse your connection. Most application will return a "host not found", or "host refused connection" error. To test, view with a web browser. If it redirects to the SD home page, you're not blocked. (using the same public IP of course)

The following conditions will cause you to be blocked. (I'll keep these numbers updated)
  • 10 attempts in two hours with an expired account will block for 2 days. (there's no real reason to let your account expire in the first place!)
  • 100 authentication failures in 2 hours will block for a day. Note, most "good" connections start with an authentication failure, but still, we're talking about 100 connection attempts in 2 hours. (was 40 in 4 hours)
  • 200 requests in 2 hours will block for 24 hours. This can include valid requests. (was 100 in 2 hours)
I realize this is somewhat subjective and I don't want to block people trying to use the service, so if you run into a block, let's discuss it and I'm willing to unblock you, and even adjust the criteria.

Note to application developers. If you get a failure, look at the status and decide what to do. If you get a bad password or expired account, there really isn't a need to try again. If you do get an error that should be retried, by all means back off an exponential time and try again (5min, 10min, 20min, etc).



Re: Problems connecting to sd-dd servers ( blocking )

Posted: Sat Nov 15, 2014 3:38 pm
by amblinn
Is there some condition where a "normal user" might become blocked unintentionally? And is there some way for us to get back on the "good" list?

Both my WiRNS and my MythTV systems are giving me "failing to connect"-type messages, and the test page also tells me that it's not reachable (unless I'm checking while I've VPN'd into my employer's system).

I'm not running anything else that might be hitting schedules direct automatically, and I haven't changed default settings on either WiRNS or MythTV.

My external IP address is assigned automatically by Comcast; I don't know how frequently it's changed.


Re: Problems connecting to sd-dd servers ( blocking )

Posted: Sat Nov 15, 2014 5:27 pm
by hisdoulos
It looks like I am blocked , i was trying to genuinely setup MythTV for the first time and may have met your block criteria unknowingly. How can i restore my service? I am still in my try period and would like to try out and make sure it works before i subscribe. Please look into this and let me know.

Re: Problems connecting to sd-dd servers ( blocking )

Posted: Sat Nov 15, 2014 10:09 pm
by rmeden
amblinn wrote:Is there some condition where a "normal user" might become blocked unintentionally? And is there some way for us to get back on the "good" list?
It's certainly possible, but not my intention. If you provide your public IP (here, PM, or email to admin@sd) I'll look into what happened.

Since you're running SD on multiple devices, do they all poll for data around the same time? Spread them out (by hours).

Re: Problems connecting to sd-dd servers ( blocking )

Posted: Sat Nov 15, 2014 10:29 pm
by rmeden
Got amblinn's IP via private message.

Based on feedback here, I've more than doubled the unexpired account limits.

The only active blocks at the moment are expired accounts, and they don't get much sympathy from me. (plus there are a lot of them!)


Re: Problems connecting to sd-dd servers ( blocking )

Posted: Sat Nov 15, 2014 10:59 pm
by amblinn
Thank you!! I'm getting the schedules again.

I don't think my two machines are polling very frequently... I have MythTV set to grab data when suggested by SD (rather than setting up a cron, or specifying the time); it seems to be about once every 20-30 hours. WiRNS normally polls every 4-6(?) hours, I think... if it fails, it appears to try 3 times (once/minute) before waiting another couple hours to try again. But, the WiRNS computer is usually turned off, so it seems odd that this would be tipping the scales. Perhaps (coincidentally) someone else had the IP address I'm using now, and managed to get it blacklisted. Or, maybe, I hit a "perfect storm" where both my machines decided to hit SD at the same time.

Either way, just happy that the data is flowing again. All things considered, this has been a relatively easy transition, from a user perspective. Thank you!

Re: Problems connecting to sd-dd servers ( blocking )

Posted: Sun Nov 16, 2014 6:49 pm
by mrburns42
I am using TVHeadend running on a Pogoplug. This was working fine until last Sunday 11/9/2014. At that time, I noticed that program data did not extend to 11/10/2014. The last data I had was for 11/9/2014. Checking my subscription, it is active. I changed the polling rate from once every 6 hours to once every 6 seconds. I did this to see if I could find any hints as to what is wrong in the log files. I found nothing useful in the logs and forgot to revert the settings.

Eventually, I found the recommended change to the host file and made that change. However, that too did not get me any data. The program TVHeadend is running a perl script to gather the data the script is tv_grab_na_dd When I attempt to execute that script manually, I get a connection refused as shown below.

Service description ' ... ivery.wsdl' can't be loaded: 500 Can't connect to (Connection refused)

Therefore, I suspect that I am being blocked. I changed my polling interval back to once in six hours, so I guess that whenever the 24 hour time limit is up, that I will at least be able to poll again.

The funny part is that will an entire week that the polling interval was once every six seconds and the host file was updated, I never got any data.

If I ping, then I get

64 bytes from ( icmp_seq=1 ttl=47 time=58.2 ms

Therefore, I think it is resolving to the correct IP address.

Anyway, I apologize for sending so many requests. I was not really aware that it would be any issue for anyone. For active subscribers, perhaps along with blocking their IP address, you could send an email informing them as to why they are blocked.

Re: Problems connecting to sd-dd servers ( blocking )

Posted: Sun Nov 16, 2014 9:34 pm
by rmeden
Best solution for you isn't to update your host file, but to update to the latest version of tv_grab_na_dd.

The old tv_grab_na_dd should still work without a hostname change (for now.. Tribune hasn't shut down DNS) so I suspect you have another issue.


Re: Problems connecting to sd-dd servers ( blocking )

Posted: Mon Nov 17, 2014 7:46 pm
by mrburns42
OK, after a 24 hour wait, I am no longer blocked. If I run tv_grab_na_dd from a console window, it executes without error and informs me that it "Downloaded 110 programs in 10 seconds". However, the TVHeadend GUI has no EPG data. I enabled logging in TVHeaded and then set the update interval back to every six seconds for about 15 seconds, then I reverted it back to every six hours. TVHeadend did not produce any errors, but it did not find any channels to parse. It tried twice and seems to have downloaded, but did not find anything that matched my channels in the data. All I can conclude is that either my channel info has mysteriously changed in TVHeadend, or the format of the data I am receiving is now different than it was previously.

It will take me some time to figure this out, as the TVHeadend box will be busy the next few nights recording shows for the wife. When she is not using it, I will dig in further to try to figure out what is different about either my channels or the data.

I am posting this update mainly as data for anyone else experiencing this same issue.

Re: Problems connecting to sd-dd servers ( blocking )

Posted: Tue Nov 18, 2014 11:43 am
by rmeden
The --output of tv_grab_na_dd (XMLTV format) has not changed.

Glad to heard it's not a download data version, but unfortunately, I don't know how to troubleshoot your app issue.