Page 1 of 3

2017-06-07: Service disruption

Posted: Wed Jun 07, 2017 7:56 pm
by rkulagow
As of 2017-06-07 @ 9:48P CDT / 2017-06-08 @ 02:40 UTC

Our upstream data provider has been experiencing issues; the data they are returning is syntactically valid, but is incorrect in various ways.

* Lineups have fewer than expected channels.
* Schedule data is either returning no data, or severely limited data.

We have opened tickets in their internal portal and have engaged their after-hours, on-call support structure.

Until the issue is resolved upstream, we will stop updating the schedules to avoid polluting the existing data with bad data.

Additional information will be added to this thread as we become aware of it.

Re: 2017-06-07: Service disruption

Posted: Wed Jun 07, 2017 8:14 pm
by rkulagow
2017-06-07 @ 10:13P CDT / 2017-06-08 @ 03:13 UTC

I have spoken to the on-call support person and they have ACKed the issue and are escalating to their API team.

Re: 2017-06-07: Service disruption

Posted: Wed Jun 07, 2017 10:15 pm
by rkulagow
2017-06-08 @ 12:13A CDT / 05:13 UTC

Issue has been ACKed as reproducible by upstream API team and investigation continues.

Re: 2017-06-07: Service disruption

Posted: Thu Jun 08, 2017 7:24 am
by rkulagow
2017-06-08 @ 09:23A CDT / 14:23 UTC

Upstream API engineers are still troubleshooting root cause. No ETA for restoration of service is available at this time.

Re: 2017-06-07: Service disruption

Posted: Thu Jun 08, 2017 12:11 pm
by rkulagow
2017-06-08 @ 02:10P CDT / 19:10 UTC

I have requested a status update from our upstream.

Re: 2017-06-07: Service disruption

Posted: Thu Jun 08, 2017 2:57 pm
by rkulagow
2017-06-08 @ 04:55P CDT / 21:55 UTC

Our upstream is troubleshooting with their CDN provider and is also putting together an action plan on a work-around to restore service.

Re: 2017-06-07: Service disruption

Posted: Fri Jun 09, 2017 4:32 am
by rkulagow
2017-06-09 @ 06:30A CDT / 11:30 UTC

I have requested a status update from our upstream.

Re: 2017-06-07: Service disruption

Posted: Fri Jun 09, 2017 8:44 am
by rkulagow
2017-06-09 @ 10:42A CDT / 15:42 UTC

Some upstream services are beginning to be restored; we will begin testing updates shortly.

Re: 2017-06-07: Service disruption

Posted: Fri Jun 09, 2017 12:49 pm
by rkulagow
2017-06-09 @ 02:48P CDT / 19:48 UTC

We are beginning to run updates against the upstream service, at a reduced rate, in order to avoid swamping their systems as systems come online.

Re: 2017-06-07: Service disruption

Posted: Fri Jun 09, 2017 1:56 pm
by rkulagow
I'm going to lock this topic so that it can be used to provide status updates. Please open a new thread for discussion of the outage and other questions. Thanks.