2017-06-07: Service disruption

[rkulagow]

2017-06-09 @ 06:54P CDT / 23:54 UTC

Schedules are continuing to be built server-side at Schedules Direct; once completed they will be available. However, due to not wanting to overwhelm the upstream, this will take longer than necessary.

In addition, the number of refreshes per day will be reduced from 4 to 2 to ensure that the refreshes that are launched run to completion.

[rkulagow]

2017-06-10 @ 09:21A CDT / 14:21 UTC

The upstream API issues continue, and it is impacting our ability to refresh schedules. Senior leadership at the upstream provider has been engaged.

In order to provide timely schedules we will reduce the number of days of data we are requesting from our upstream until their issue is resolved.

[rkulagow]

2017-06-10 @ 11:08A CDT / 16:08 UTC

We have reduced the number of days of schedule that we're requesting from our upstream and the bulk load on our servers is now running faster. Based on the current run-rate, ETA for schedule completion is 12:30P CDT / 17:30 UTC.

[rkulagow]

2017-06-10 @ 12:30P CDT / 17:30 UTC

The 7-day refresh has completed. Efforts are now underway to increase the amount of schedule data available to subscribers.

If you not receiving at least 7 days of schedule, please open a ticket at the Schedules Direct website so that we can investigate further. Provide the stationID in your initial ticket.

[rkulagow]

2017-06-10 @ 10:54P CDT / 2017-06-11 03:54 UTC

We are continuing to work with our upstream and will begin extending the amount of data requested in a controlled fashion to ensure that accurate schedules are received.

[rkulagow]

6/12 @ 11:51A CDT

We will continue to provide 7 days of schedule for the time being until our upstream resolves their capacity issue.

[rkulagow]

2017-06-12 @ 9:12P CDT / 2017-06-13 @ 02:12 UTC

Out upstream's engineering team is evaluating a fix to the root cause. We have been told that we'll get a status on 2017-06-13.

[rkulagow]

2017-06-13 @ 1:52P CDT / 18:52 UTC

Our upstream will be deploying a bug fix this evening to fix a query which has been generating cache misses and therefore increasing load.

[rkulagow]

2017-06-14 @ 1:30P / 18:30 UTC

Our upstream states that they applied the fix to their servers on the evening of 2017-06-13; we are evaluating if their fix has resolved the issue.

[rkulagow]

2017-06-14 @ 5:35P CDT / 2017-06-14 @ 22:35 UTC

We have run a test and now have pulled down 10 days of schedule. We will be increasing the amount of data we request over the next few days until we are satisfied that the root cause has been resolved.