Degraded service - Kallidus suite

» View Event Details | Created Mon, 04 Mar 2024 11:52:00 +0000

Post-Mortem

RCA: On 04/03/2024 between 11:45 GMT and 12:40 GMT, users may have experienced slow responsiveness or errors in Learn and other applications. The root cause was identified as poorly designed e-learning courses which were overloading our backend services and databases. We cleared the queue of requests and disabled the offending courses to restore service, and are working with the course owners to prevent future occurrences. We have also identified some improvements we can make to the course tracking process to stop this happening in future, and these are in progress.

Posted: Wed, 13 Mar 2024 11:18:00 +0000

Resolved

Products in the Kallidus Suite should now be loading correctly and site speeds have returned to expected levels.

Please accept our apologies for the inconvenience caused today.

Posted: Mon, 04 Mar 2024 12:44:00 +0000

Investigating

We are currently experiencing periods of slowness or issues accessing products in the Kallidus Suite. Our engineers are treating as high priority and we hope to have access restored to normal levels as soon as possible.

Please accept our apologies for the inconvenience this is currently causing.

Posted: Mon, 04 Mar 2024 11:52:00 +0000

Degraded service - Kallidus Suite

» View Event Details | Created Tue, 05 Mar 2024 11:10:00 +0000

Post-Mortem

RCA: On 05/03/2024 10:00 GMT until 12:15 GMT, users may experienced slow responsiveness or errors from all Kallidus applications. This was caused by exceptionally high Learn usage during the morning, creating a bottleneck in the databases and leading to slow responses from shared services. Queued requests were cleared to clear the bottleneck and restore service, and mitigation measures were taken to ensure the excessive load would not reoccur. Analysis and improvements to the affected services is continuing, including measures to prevent high load on the databases in the future.

Posted: Wed, 13 Mar 2024 10:04:00 +0000

Resolved

Our Engineers have resolved the access and responsiveness issues in the Kallidus Suite. An RCA for this incident will be posted here in due course.

Apologies again for the inconvenience this caused today.

Posted: Tue, 05 Mar 2024 12:08:00 +0000

Update

Our Engineers are continuing to investigate the cause of intermittent site unresponsiveness and slow page response times.

Please accept our apologies for the continued inconvenience this is causing today.

Posted: Tue, 05 Mar 2024 12:07:00 +0000

Investigating

We are currently experiencing intermittent periods of slow response times across products in the Kallidus Suite. Our engineers are currently investigating and we hope to have services restored to normal shortly.

Posted: Tue, 05 Mar 2024 11:10:00 +0000

Degraded service: Kallidus Perform

» View Event Details | Created Tue, 12 Mar 2024 10:36:00 +0000

Post-Mortem

RCA: At approximately 10:05 on 12th March, some areas of Kallidus Perform became unavailable due to an update to an underlying service that supports the development activities feature. Following observations from our internal monitoring and diagnostics, this change was rolled back at 10:40. During this period customers using development activities or review forms would have experienced degraded service. Once service was fully restored, affected customers had full use of Perform again. Subsequent investigations determined that a key piece of configuration had not been updated, leading to communication errors between the service and it's persistence store. Process documentation has been updated and processes put in place to ensure configuration is synchronized and confirmed. We apologise for the inconvenience caused.

Posted: Wed, 13 Mar 2024 10:02:00 +0000

Resolved

Our engineers have now resolved the access issue for some elements of Kallidus Perform and full access should be restored now to all areas of the product. An RCA for this incident will be posted here in due course.

Please accept our apologies for the inconvenience caused.

Posted: Tue, 12 Mar 2024 11:24:00 +0000

Investigating

We are currently investigating elements of the Perform product being inaccessible at the moment.

Our engineers are investigating and we will post updates here as soon as we have identified the cause.

Posted: Tue, 12 Mar 2024 10:36:00 +0000

Degraded service: Kallidus Sapling

» View Event Details | Created Thu, 14 Mar 2024 08:24:00 +0000

Post-Mortem

RCA: Between 01:00 and 08:30 GMT on 14-03-2024, users of Kallidus HR would have experienced issues logging into Sapling HR UK.

Preliminary root cause: Kallidus engineers discovered that a database had grown sufficiently close to capacity that it had become read-only, preventing further changes from being committed. Increasing the storage space allocated to the database resolved the issue. Kallidus has a significant monitoring platform and it was discovered that the recently commissioned database had not been added to this as a result of human error. Kallidus is in the process of tightening its processes around monitoring/alerting and apologises for any inconvenience caused by this disruption.

Posted: Fri, 15 Mar 2024 15:53:00 +0000

Resolved

Our engineers have now identified the access issue for our UK hosted Sapling customers and access should be restored now. If you are still facing any login issues, please close all browsers and re-attempt signing in under a fresh browser instance. An RCA will be posted for this incident here in due course.

Once again, apologies for the inconvenience this caused.

Posted: Thu, 14 Mar 2024 08:44:00 +0000

Investigating

We are aware of an issue access Sapling UK hosted products this morning. Our engineers are investigating as priority and we will keep you updated via this status item as soon as we know more.

Please accept our apologies for the inconvenience this is causing today.

Posted: Thu, 14 Mar 2024 08:24:00 +0000

Degraded Service: Kallidus Suite

» View Event Details | Created Mon, 25 Mar 2024 11:51:00 +0000

Post-Mortem

RCA: Between 11:23 and 11:58 GMT on 25-03-2024, users of Kallidus suite may have experienced slowness when accessing the system. Preliminary root cause: Kallidus engineers discovered that a high increase of service requests negatively impacted site performance. Kallidus have identified some potential areas of improvement and are continuing to develop these changes to help reduce unnecessary web traffic. Kallidus apologises for any inconvenience caused by this disruption and thanks you for patience while we resolved the issue.

Posted: Thu, 04 Apr 2024 13:58:00 +0000

Resolved

Update: Products in the Kallidus Suite are back operational and loading normally for any affected customers.

An RCA for this incident will be made available in due course.

Please accept our apologies for any inconvenience caused.

Posted: Mon, 25 Mar 2024 12:51:00 +0000

Investigating

We are currently investigating issues of slowness or inability to access products across the Kallidus Suite for a subset of customers. Our engineers are investigating. Updates will be posted here as soon as they are made available.

Please accept our apologies for any inconvenience caused.

Posted: Mon, 25 Mar 2024 11:51:00 +0000