At approximately 10am on November 1st, part of a routine CMS deployment caused some degraded performance and page loading issues for a subset of our Prospect hosted websites. The product team were able to identify which server set was experiencing impacted performance and rebooted this to mitigate the impact, before we were able to confirm the root cause and a hotfix was under development at 11:56am and deployed at 12:06pm.
Some further performance issues were reported later at 13:56pm which were due to further CMS deployments, which were resolved at 15:17pm with a further hotfix being deployed by the product team.
The root cause was identified as being an issue with the deployment process for updates made by the Product and Infrastructure teams. The necessary fixes were planned into upcoming work sprints to ensure that the long-term solutions put in place resolved the issue causing the outage.
These fixes have been deployed and should prevent any reoccurrence of this issue.