As of 2017-06-07 @ 9:48P CDT / 2017-06-08 @ 02:40 UTC
Our upstream data provider has been experiencing issues; the data they are returning is syntactically valid, but is incorrect in various ways.
* Lineups have fewer than expected channels.
* Schedule data is either returning no data, or severely limited data.
We have opened tickets in their internal portal and have engaged their after-hours, on-call support structure.
Until the issue is resolved upstream, we will stop updating the schedules to avoid polluting the existing data with bad data.
Additional information will be added to this thread as we become aware of it.
2017-06-07: Service disruption
Re: 2017-06-07: Service disruption
2017-06-07 @ 10:13P CDT / 2017-06-08 @ 03:13 UTC
I have spoken to the on-call support person and they have ACKed the issue and are escalating to their API team.
I have spoken to the on-call support person and they have ACKed the issue and are escalating to their API team.
Re: 2017-06-07: Service disruption
2017-06-08 @ 12:13A CDT / 05:13 UTC
Issue has been ACKed as reproducible by upstream API team and investigation continues.
Issue has been ACKed as reproducible by upstream API team and investigation continues.
Re: 2017-06-07: Service disruption
2017-06-08 @ 09:23A CDT / 14:23 UTC
Upstream API engineers are still troubleshooting root cause. No ETA for restoration of service is available at this time.
Upstream API engineers are still troubleshooting root cause. No ETA for restoration of service is available at this time.
Re: 2017-06-07: Service disruption
2017-06-08 @ 02:10P CDT / 19:10 UTC
I have requested a status update from our upstream.
I have requested a status update from our upstream.
Re: 2017-06-07: Service disruption
2017-06-08 @ 04:55P CDT / 21:55 UTC
Our upstream is troubleshooting with their CDN provider and is also putting together an action plan on a work-around to restore service.
Our upstream is troubleshooting with their CDN provider and is also putting together an action plan on a work-around to restore service.
Re: 2017-06-07: Service disruption
2017-06-09 @ 06:30A CDT / 11:30 UTC
I have requested a status update from our upstream.
I have requested a status update from our upstream.
Re: 2017-06-07: Service disruption
2017-06-09 @ 10:42A CDT / 15:42 UTC
Some upstream services are beginning to be restored; we will begin testing updates shortly.
Some upstream services are beginning to be restored; we will begin testing updates shortly.
Re: 2017-06-07: Service disruption
2017-06-09 @ 02:48P CDT / 19:48 UTC
We are beginning to run updates against the upstream service, at a reduced rate, in order to avoid swamping their systems as systems come online.
We are beginning to run updates against the upstream service, at a reduced rate, in order to avoid swamping their systems as systems come online.
Re: 2017-06-07: Service disruption
I'm going to lock this topic so that it can be used to provide status updates. Please open a new thread for discussion of the outage and other questions. Thanks.