2022-05-21T03:00Z - migration started.
2022-05-21T04:55Z - migration ended.
2022-05-21T22:00Z - caching logic updated to reduce number of database requests; the database server is running out of connections, so some invocations are failing and returning "Internal Server Error".
2022-05-22T02:00Z - migration to larger database servers started. Under load, the database server which had been working in the test environment is not able to maintain throughput, so the process to move the data to a new server has begun. Until the new server is online, users may continue to experience issues in retrieving a token; if possible, increase the timeout for token retrieval to 29 seconds. The logs are indicating that the token response is taking in excess of 4 seconds due to the code not being able to obtain a database handle on first attempt.
2022-05-22T02:45Z - to complete the migration of data to new servers, the service will be placed into Offline mode.
2022-05-22T05:40Z - migration of data is continuing. Indexes are being rebuilt.
2022-05-22T14:39Z - indexes rebuilt. Data ingest started to catch up to data which was refreshed upstream but which was not processed due to database overload.
2022-05-22T15:10Z - precaching of user lineups and stationsIDs into Redis. This is normally performed just-in-time from the database, but the influx of users when the service is put back online will cause database load.
2022-05-22T16:07Z - AWS Database Migration Service mis-identified primary and unique keys and did not transfer all data from source database to destination. Analyzing which tables are affected.
2022-05-22T19:02Z - Tables which were improperly migrated and the relevant data has been rebuilt.
2022-05-22T21:00Z - Service was re-enabled.
2022-05-22T21:35Z - Lambda concurrency limit reached; request made to increase concurrent execution limits.
2022-05-22T22:54Z - Increased max_allowed_packet.
2022-05-22T23:30Z - Reduced concurrency limit for a particular endpoint to make invocations available to issue_token and mainline endpoints.
2022-05-23T04:00Z - Service is being migrated back to existing PHP API until root cause of failure to obtain read handle against database can be determined. Python backend placed into offline mode.
2022-05-23T04:20Z - PHP API updating per normal daily update; when PHP service has 'caught up' to 3 days of non-updates, it will be placed back into service.
2022-05-23T06:53Z - Existing servers placed back into service.
What should you expect?
- The Python provided API has been created to be protocol compatible, so it's likely that you won't need to do anything if your grabber has implemented the published API.
- The new service is provided by AWS API Gateway, so the issues you may have experienced if you were updating your schedule at your local midnight time should now be addressed.
- The methods used to generate data on our backend servers have been updated, and there are a number of fields in the new JSON which will cause the MD5 values to change, so your application will refresh its local data. This is known as a cache invalidation - expect that there will be a large amount of data downloaded.
- The schedules on the backend will be refreshed more often; we have migrated to a new data delivery mechanism from our upstream provider.
- You should see 14 to 18 days of schedule data.
- Images will be updated more often.
- Lineup updates will be occur more often.
- Schedules Direct will send lineup deletion notifications to your registered email address 30 days before a lineup is deleted, in order to give you time to migrate. Lineups are deleted by upstream providers if they're consolidating headends, or ending an offering - this typically occurs if a provider is migrating from analog (usually, the -DEFAULT lineup) to digital-only (-X)
- Check to see if you're running the latest version of your application / grabber. Developers were provided early access to the new delivery mechanism, and a number of issues were resolved.
- Check the support forum for your application and engage the developer; they'll be the ones that can help you troubleshoot the application and the migration.
- If you're still stuck, your developer will tell you to open a ticket with Schedules Direct. Please open a lineup support ticket on the SchedulesDirect webpage https://www.schedulesdirect.org/lineupsupport and indicate that you're having an issue, and that the developer of your application has directed you to open a ticket.