Summary

Incident Start time: 12:34pm

Incident End Time: 12:50pm

Business Services Impacted: CRM Login and Service

The Microsoft ASP (Application Service Plan) instance stopped responding, causing any queries being sent to the Prospect OData Lite API to not reach the server and get stuck in pending, eventually timing out. This meant any users attempting to login would not be able to gain access to the CRM, and any logged in users were not able to navigate to records or refresh any open CRM records, preventing them from using the CRM. The incident was quickly identified by the support engineers and raised with the Product engineers to investigate further while an a Status Page incident was posted. The Product Team identified the root cause and implemented a fix that restored service and CRM requests were completing successfully once the fix was deployed.

Timeline

At approximately 12:34pm on 3/10/23, requests being sent to OData Lite stopped being completed, and Prospect CRM became unresponsive.

This was alerted both inhouse and by users, and raised as an priority issue with the Product team to investigate by 12:40pm.

Head of SRE identified the ASP (Application Service Plan) instance is not responding at 12:50pm.

The ASP instance is scaled-up and redeployed at 12:50pm.

This restored normal service, requests begin to complete successfully and users are able to resume using the CRM.