Page 1 of 1

2020-09-22 Service Outage

Posted: Tue Sep 22, 2020 9:22 pm
by rkulagow
On 2020-09-22, there was a service outage on the JSON service. This was caused due to backend changes required to support a data feed for Australia.

A side-effect of the changes was that the stationID within the JSON data did not meet the published specification, which states that the stationID is a string, and not an integer. For approximately 12h, as schedule data expired and was updated, the updates contained the stationID in the incorrect format.

Once the issue was identified, a fix was deployed. However, a processing node did not receive the updated code, so it continued to generate schedules in the wrong format. In order to remove the bad data, all processing nodes were rebuilt, and schedules were re-generated.