2020-09-22 Service Outage

Post Reply
rkulagow
SD Staff
Posts: 933
Joined: Tue Aug 14, 2007 3:15 pm

2020-09-22 Service Outage

Post by rkulagow »

On 2020-09-22, there was a service outage on the JSON service. This was caused due to backend changes required to support a data feed for Australia.

A side-effect of the changes was that the stationID within the JSON data did not meet the published specification, which states that the stationID is a string, and not an integer. For approximately 12h, as schedule data expired and was updated, the updates contained the stationID in the incorrect format.

Once the issue was identified, a fix was deployed. However, a processing node did not receive the updated code, so it continued to generate schedules in the wrong format. In order to remove the bad data, all processing nodes were rebuilt, and schedules were re-generated.

Post Reply