2017-06-07: Service disruption

Re: 2017-06-07: Service disruption

Postby rkulagow » Fri Jun 09, 2017 4:57 pm

2017-06-09 @ 06:54P CDT / 23:54 UTC

Schedules are continuing to be built server-side at Schedules Direct; once completed they will be available. However, due to not wanting to overwhelm the upstream, this will take longer than necessary.

In addition, the number of refreshes per day will be reduced from 4 to 2 to ensure that the refreshes that are launched run to completion.
rkulagow
SD Staff
 
Posts: 896
Joined: Tue Aug 14, 2007 3:15 pm

Re: 2017-06-07: Service disruption

Postby rkulagow » Sat Jun 10, 2017 7:24 am

2017-06-10 @ 09:21A CDT / 14:21 UTC

The upstream API issues continue, and it is impacting our ability to refresh schedules. Senior leadership at the upstream provider has been engaged.

In order to provide timely schedules we will reduce the number of days of data we are requesting from our upstream until their issue is resolved.
rkulagow
SD Staff
 
Posts: 896
Joined: Tue Aug 14, 2007 3:15 pm

Re: 2017-06-07: Service disruption

Postby rkulagow » Sat Jun 10, 2017 9:11 am

2017-06-10 @ 11:08A CDT / 16:08 UTC

We have reduced the number of days of schedule that we're requesting from our upstream and the bulk load on our servers is now running faster. Based on the current run-rate, ETA for schedule completion is 12:30P CDT / 17:30 UTC.
rkulagow
SD Staff
 
Posts: 896
Joined: Tue Aug 14, 2007 3:15 pm

Re: 2017-06-07: Service disruption

Postby rkulagow » Sat Jun 10, 2017 10:33 am

2017-06-10 @ 12:30P CDT / 17:30 UTC

The 7-day refresh has completed. Efforts are now underway to increase the amount of schedule data available to subscribers.

If you not receiving at least 7 days of schedule, please open a ticket at the Schedules Direct website so that we can investigate further. Provide the stationID in your initial ticket.
rkulagow
SD Staff
 
Posts: 896
Joined: Tue Aug 14, 2007 3:15 pm

Re: 2017-06-07: Service disruption

Postby rkulagow » Sat Jun 10, 2017 8:56 pm

2017-06-10 @ 10:54P CDT / 2017-06-11 03:54 UTC

We are continuing to work with our upstream and will begin extending the amount of data requested in a controlled fashion to ensure that accurate schedules are received.
rkulagow
SD Staff
 
Posts: 896
Joined: Tue Aug 14, 2007 3:15 pm

Re: 2017-06-07: Service disruption

Postby rkulagow » Mon Jun 12, 2017 9:52 am

6/12 @ 11:51A CDT

We will continue to provide 7 days of schedule for the time being until our upstream resolves their capacity issue.
rkulagow
SD Staff
 
Posts: 896
Joined: Tue Aug 14, 2007 3:15 pm

Re: 2017-06-07: Service disruption

Postby rkulagow » Mon Jun 12, 2017 7:15 pm

2017-06-12 @ 9:12P CDT / 2017-06-13 @ 02:12 UTC

Out upstream's engineering team is evaluating a fix to the root cause. We have been told that we'll get a status on 2017-06-13.
rkulagow
SD Staff
 
Posts: 896
Joined: Tue Aug 14, 2007 3:15 pm

Re: 2017-06-07: Service disruption

Postby rkulagow » Tue Jun 13, 2017 11:54 am

2017-06-13 @ 1:52P CDT / 18:52 UTC

Our upstream will be deploying a bug fix this evening to fix a query which has been generating cache misses and therefore increasing load.
rkulagow
SD Staff
 
Posts: 896
Joined: Tue Aug 14, 2007 3:15 pm

Re: 2017-06-07: Service disruption

Postby rkulagow » Wed Jun 14, 2017 11:31 am

2017-06-14 @ 1:30P / 18:30 UTC

Our upstream states that they applied the fix to their servers on the evening of 2017-06-13; we are evaluating if their fix has resolved the issue.
rkulagow
SD Staff
 
Posts: 896
Joined: Tue Aug 14, 2007 3:15 pm

Re: 2017-06-07: Service disruption

Postby rkulagow » Wed Jun 14, 2017 3:38 pm

2017-06-14 @ 5:35P CDT / 2017-06-14 @ 22:35 UTC

We have run a test and now have pulled down 10 days of schedule. We will be increasing the amount of data we request over the next few days until we are satisfied that the root cause has been resolved.
rkulagow
SD Staff
 
Posts: 896
Joined: Tue Aug 14, 2007 3:15 pm

PreviousNext

Return to Service Issues

Who is online

Users browsing this forum: No registered users and 1 guest