The cause was an update to our report execution system to make the report status more legible, this caused an issue with retrying reports which caused many unnecessary report executions which blocked the queue of reports. We will implement better testing for the report retry logic.
Posted Jul 15, 2024 - 08:03 UTC
Monitoring
A fix has been implemented and and the backlog of reports has now finished processing.
Posted Jul 15, 2024 - 07:04 UTC
Identified
This issue has been identified and a fix is being prepared.
Posted Jul 15, 2024 - 06:30 UTC
Investigating
Reports are currently stuck in processing or not being executed.
Posted Jul 15, 2024 - 06:20 UTC
This incident affected: Administration Backend Server.