2017-06-07: Service disruption

2017-06-07: Service disruption

Postby rkulagow » Wed Jun 07, 2017 7:56 pm

As of 2017-06-07 @ 9:48P CDT / 2017-06-08 @ 02:40 UTC

Our upstream data provider has been experiencing issues; the data they are returning is syntactically valid, but is incorrect in various ways.

* Lineups have fewer than expected channels.
* Schedule data is either returning no data, or severely limited data.

We have opened tickets in their internal portal and have engaged their after-hours, on-call support structure.

Until the issue is resolved upstream, we will stop updating the schedules to avoid polluting the existing data with bad data.

Additional information will be added to this thread as we become aware of it.
rkulagow
SD Staff
 
Posts: 880
Joined: Tue Aug 14, 2007 3:15 pm

Re: 2017-06-07: Service disruption

Postby rkulagow » Wed Jun 07, 2017 8:14 pm

2017-06-07 @ 10:13P CDT / 2017-06-08 @ 03:13 UTC

I have spoken to the on-call support person and they have ACKed the issue and are escalating to their API team.
rkulagow
SD Staff
 
Posts: 880
Joined: Tue Aug 14, 2007 3:15 pm

Re: 2017-06-07: Service disruption

Postby rkulagow » Wed Jun 07, 2017 10:15 pm

2017-06-08 @ 12:13A CDT / 05:13 UTC

Issue has been ACKed as reproducible by upstream API team and investigation continues.
rkulagow
SD Staff
 
Posts: 880
Joined: Tue Aug 14, 2007 3:15 pm

Re: 2017-06-07: Service disruption

Postby rkulagow » Thu Jun 08, 2017 7:24 am

2017-06-08 @ 09:23A CDT / 14:23 UTC

Upstream API engineers are still troubleshooting root cause. No ETA for restoration of service is available at this time.
rkulagow
SD Staff
 
Posts: 880
Joined: Tue Aug 14, 2007 3:15 pm

Re: 2017-06-07: Service disruption

Postby rkulagow » Thu Jun 08, 2017 12:11 pm

2017-06-08 @ 02:10P CDT / 19:10 UTC

I have requested a status update from our upstream.
rkulagow
SD Staff
 
Posts: 880
Joined: Tue Aug 14, 2007 3:15 pm

Re: 2017-06-07: Service disruption

Postby rkulagow » Thu Jun 08, 2017 2:57 pm

2017-06-08 @ 04:55P CDT / 21:55 UTC

Our upstream is troubleshooting with their CDN provider and is also putting together an action plan on a work-around to restore service.
rkulagow
SD Staff
 
Posts: 880
Joined: Tue Aug 14, 2007 3:15 pm

Re: 2017-06-07: Service disruption

Postby rkulagow » Fri Jun 09, 2017 4:32 am

2017-06-09 @ 06:30A CDT / 11:30 UTC

I have requested a status update from our upstream.
rkulagow
SD Staff
 
Posts: 880
Joined: Tue Aug 14, 2007 3:15 pm

Re: 2017-06-07: Service disruption

Postby rkulagow » Fri Jun 09, 2017 8:44 am

2017-06-09 @ 10:42A CDT / 15:42 UTC

Some upstream services are beginning to be restored; we will begin testing updates shortly.
rkulagow
SD Staff
 
Posts: 880
Joined: Tue Aug 14, 2007 3:15 pm

Re: 2017-06-07: Service disruption

Postby rkulagow » Fri Jun 09, 2017 12:49 pm

2017-06-09 @ 02:48P CDT / 19:48 UTC

We are beginning to run updates against the upstream service, at a reduced rate, in order to avoid swamping their systems as systems come online.
rkulagow
SD Staff
 
Posts: 880
Joined: Tue Aug 14, 2007 3:15 pm

Re: 2017-06-07: Service disruption

Postby rkulagow » Fri Jun 09, 2017 1:56 pm

I'm going to lock this topic so that it can be used to provide status updates. Please open a new thread for discussion of the outage and other questions. Thanks.
rkulagow
SD Staff
 
Posts: 880
Joined: Tue Aug 14, 2007 3:15 pm

Next

Return to Service Issues

Who is online

Users browsing this forum: No registered users and 1 guest