Customer Websites Degraded Performance

Incident Report for Prospect CRM's

Postmortem

At approximately 10am on November 1st, part of a routine CMS deployment caused some degraded performance and page loading issues for a subset of our Prospect hosted websites. The product team were able to identify which server set was experiencing impacted performance and rebooted this to mitigate the impact, before we were able to confirm the root cause and a hotfix was under development at 11:56am and deployed at 12:06pm.

Some further performance issues were reported later at 13:56pm which were due to further CMS deployments, which were resolved at 15:17pm with a further hotfix being deployed by the product team.

The root cause was identified as being an issue with the deployment process for updates made by the Product and Infrastructure teams. The necessary fixes were planned into upcoming work sprints to ensure that the long-term solutions put in place resolved the issue causing the outage.

These fixes have been deployed and should prevent any reoccurrence of this issue.

Posted Jan 15, 2025 - 15:02 GMT

Resolved

This incident has been resolved.
Posted Nov 01, 2024 - 16:25 GMT

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Nov 01, 2024 - 15:17 GMT

Investigating

Following the server restarts earlier which resolved some of the speed and downtime issues, some CMS sites are now experiencing further outages. We are continuing to investigate the issue.
Posted Nov 01, 2024 - 13:55 GMT

Monitoring

The hotfix has been deployed and this appears to have significantly reduced load times and timeout messages when loading up hosted websites, we are continuing to monitor to ensure that this has been fully resolved.
Posted Nov 01, 2024 - 12:06 GMT

Update

We have identified a possible root cause and are working on coding and deploying a hotfix as soon as possible.
Posted Nov 01, 2024 - 11:56 GMT

Update

The server sets have successfully rebooted and slightly improved website response time, but we are continuing to investigate to find a full resolution.
Posted Nov 01, 2024 - 11:02 GMT

Update

We will shortly be rebooting the server sets running the CMS hosting platform to resolve the issue.
Posted Nov 01, 2024 - 10:44 GMT

Identified

The issue has been identified and a fix is being implemented.
Posted Nov 01, 2024 - 10:37 GMT

Investigating

We are currently investigating an issue related to customer websites responding slower than expected or timing out. We will provide further updates as information becomes available.
Posted Nov 01, 2024 - 10:12 GMT
This incident affected: Our Website and Individual Prospect Apps (Prospect CMS Editor App, Hosted Customer Websites, Portals and eCommerce sites).