Degraded performance of Prospect CRM

Incident Report for Prospect CRM's

Postmortem

At approximately 9:50am on Thursday 4/7 an instance was recycled due to a memory limit being reached, causing degraded CRM performance for a subset of workspaces. We scaled these instances up to double the available memory, and scaled this out to add an additional instance for better load balancing, this was applied at 9:55am, and with a short delay as the instances were “warmed up”, regular CRM perfomance was restored by 10:00am.

Following a period of monitoring and reflection, we have re-scaled the instances in, removing the additional instance added, but have continued to keep the instances scaled up to avoid hitting any further memory limits and preventing further performance issues.

Posted Jul 09, 2024 - 11:59 BST

Resolved

This incident has been resolved, the services will remained scaled-up while we continue our post-mortem investigation, and to prevent any further reoccurrences.

Posted Jul 04, 2024 - 11:30 BST

Monitoring

The upscaled instances have restored regular performance, and we are continuing to monitor the situation.

Posted Jul 04, 2024 - 10:40 BST

Update

We are applying some mitigation, this may cause a brief blip while the scaled instances warm-up.

Posted Jul 04, 2024 - 09:58 BST

Identified

Some customers have reported slower than expected performance of Prospect CRM and associated services. We are currently investigating and will provide further updates as information becomes available.

Posted Jul 04, 2024 - 09:57 BST

This incident affected: Individual Prospect Apps (Prospect CRM App) and APIs, Login, and Core Platform.