2017-06-07: Service disruption

rkulagow
SD Staff
Posts: 933
Joined: Tue Aug 14, 2007 3:15 pm

2017-06-07: Service disruption

Post by rkulagow »

As of 2017-06-07 @ 9:48P CDT / 2017-06-08 @ 02:40 UTC

Our upstream data provider has been experiencing issues; the data they are returning is syntactically valid, but is incorrect in various ways.

* Lineups have fewer than expected channels.
* Schedule data is either returning no data, or severely limited data.

We have opened tickets in their internal portal and have engaged their after-hours, on-call support structure.

Until the issue is resolved upstream, we will stop updating the schedules to avoid polluting the existing data with bad data.

Additional information will be added to this thread as we become aware of it.

rkulagow
SD Staff
Posts: 933
Joined: Tue Aug 14, 2007 3:15 pm

Re: 2017-06-07: Service disruption

Post by rkulagow »

2017-06-07 @ 10:13P CDT / 2017-06-08 @ 03:13 UTC

I have spoken to the on-call support person and they have ACKed the issue and are escalating to their API team.

rkulagow
SD Staff
Posts: 933
Joined: Tue Aug 14, 2007 3:15 pm

Re: 2017-06-07: Service disruption

Post by rkulagow »

2017-06-08 @ 12:13A CDT / 05:13 UTC

Issue has been ACKed as reproducible by upstream API team and investigation continues.

rkulagow
SD Staff
Posts: 933
Joined: Tue Aug 14, 2007 3:15 pm

Re: 2017-06-07: Service disruption

Post by rkulagow »

2017-06-08 @ 09:23A CDT / 14:23 UTC

Upstream API engineers are still troubleshooting root cause. No ETA for restoration of service is available at this time.

rkulagow
SD Staff
Posts: 933
Joined: Tue Aug 14, 2007 3:15 pm

Re: 2017-06-07: Service disruption

Post by rkulagow »

2017-06-08 @ 02:10P CDT / 19:10 UTC

I have requested a status update from our upstream.

rkulagow
SD Staff
Posts: 933
Joined: Tue Aug 14, 2007 3:15 pm

Re: 2017-06-07: Service disruption

Post by rkulagow »

2017-06-08 @ 04:55P CDT / 21:55 UTC

Our upstream is troubleshooting with their CDN provider and is also putting together an action plan on a work-around to restore service.

rkulagow
SD Staff
Posts: 933
Joined: Tue Aug 14, 2007 3:15 pm

Re: 2017-06-07: Service disruption

Post by rkulagow »

2017-06-09 @ 06:30A CDT / 11:30 UTC

I have requested a status update from our upstream.

rkulagow
SD Staff
Posts: 933
Joined: Tue Aug 14, 2007 3:15 pm

Re: 2017-06-07: Service disruption

Post by rkulagow »

2017-06-09 @ 10:42A CDT / 15:42 UTC

Some upstream services are beginning to be restored; we will begin testing updates shortly.

rkulagow
SD Staff
Posts: 933
Joined: Tue Aug 14, 2007 3:15 pm

Re: 2017-06-07: Service disruption

Post by rkulagow »

2017-06-09 @ 02:48P CDT / 19:48 UTC

We are beginning to run updates against the upstream service, at a reduced rate, in order to avoid swamping their systems as systems come online.

rkulagow
SD Staff
Posts: 933
Joined: Tue Aug 14, 2007 3:15 pm

Re: 2017-06-07: Service disruption

Post by rkulagow »

I'm going to lock this topic so that it can be used to provide status updates. Please open a new thread for discussion of the outage and other questions. Thanks.

Locked