Degraded service: Kallidus Learn

» View Event Details | Created Thu, 04 Apr 2024 07:02:00 +0000

Post-Mortem

RCA: From 08:00 BST to 08:10 BST on 4th April customers would have been unable to launch eLearning content. This was detected during our post-deployment testing but we were unable to rectify the issue before the end of the maintenance window. It was caused by a missing configuration item for some functionality that is to be released in the future, and normal operation was restored once the configuration item had been added. We will review this change to see we can improve this process, and how we can ensure testing and resolution can be completed within the maintenance window. We apologise to any customers that may have experienced the issue this morning.

Posted: Thu, 04 Apr 2024 09:50:00 +0000

Resolved

Our engineers have now resolved issues with launching e-learning from the student side of Learn following this morning's maintenance and Learn upgrade.

An RCA for this incident will be posted here in due course.

Please accept our apologies for any inconvenience caused.

Posted: Thu, 04 Apr 2024 07:16:00 +0000

Investigating

Our usual Thursday maintenance window is currently overrunning and customers are currently seeing an issue with launching e-learning titles from within Learn. Our engineers are currently investigating and we hope to have the matter resolved as soon as possible.

Posted: Thu, 04 Apr 2024 07:02:00 +0000

Degraded service: Kallidus Suite

» View Event Details | Created Mon, 15 Apr 2024 14:29:00 +0000

Post-Mortem

Between 10:34 and 16:04 GMT on 15-04-2024, users of Kallidus suite may have experienced intermittent slowness when accessing the system.

Preliminary root cause: Kallidus engineers discovered an increase in response times for services providing identity for the Kallidus suite, this had a negative impact on overall response times. Kallidus engineers identified potential scaling improvements. Once implemented, services returned to normal. Kallidus apologises for any inconvenience caused by this disruption.

Between 10:30 and 11:13 GMT on 16-04-2024, users of Kallidus suite may have experienced slowness when accessing the system.

Preliminary root cause: Kallidus engineers discovered an increase in response times the Kallidus suite. Routing services were reloaded to return site speed to normal. Improvements to monitoring have been implemented to help our continued investigations into these issues. Kallidus apologises for any inconvenience caused by this disruption.

Posted: Mon, 22 Apr 2024 10:59:00 +0000

Resolved

Good service has now resumed and we have seen good loading page times across our Kallidus Suite for the afternoon. Monitoring is still in place and an RCA for this incident will be made available here in due course.

Posted: Tue, 16 Apr 2024 15:58:00 +0000

Update

Update: Our engineers are continuing to monitor our services and Suite products. We are seeing normal levels of access and speed to load pages at this time. Apologies for any inconvenience that this morning's slowness caused to affected customers.

Posted: Tue, 16 Apr 2024 14:34:00 +0000

Update

Update: Page load times have improved significantly and various mitigating steps have been put in place to alleviate the issues we've been seeing with site access speeds today. The team are continuing to monitor the situation and we will be keeping this alert open as an active incident until we're confident it's fully resolved.

Posted: Tue, 16 Apr 2024 11:54:00 +0000

Update

Engineers are continuing to investigate reports of Kallidus Suite slowness in Kallidus products across our Suite for a sub set of customers. Please accept our apologies for the continued inconvenience caused.

Posted: Tue, 16 Apr 2024 10:54:00 +0000

Update

Engineers are continuing to investigate further reports of Kallidus Suite slowness to login and access pages in Kallidus products across our Suite for a sub set of customers.

Please accept our apologies for the continued inconvenience caused.

Posted: Tue, 16 Apr 2024 09:36:00 +0000

Update

Our engineers have resolved the immediate issues with slowness to our Kallidus Suite and we are seeing page load speeds back to normal levels.

Posted: Tue, 16 Apr 2024 06:55:00 +0000

Update

Engineers are continuing to investigate the cause of the Kallidus suite slowness. Page load times have improved significantly since the latest post, but we are keeping this status item open until we have fully investigated and resolved. Thank you for your patience while we continue to investigate.

Posted: Mon, 15 Apr 2024 15:31:00 +0000

Investigating

Degraded service: Kallidus Suite Our engineers are currently investigating slowness to access login and pages within the Kallidus Suite. We will continue to update this page as our teams work to resolve the issues. Please accept our apologies for the inconvenience this is currently causing.

Posted: Mon, 15 Apr 2024 14:29:00 +0000

Learn Disruption

» View Event Details | Created Mon, 22 Apr 2024 09:42:00 +0000

Post-Mortem

RCA: Between 10:00 and 10:44 BST on 22/04/2024 Learn may have been unresponsive for some customers (indicated by a continuous spinning circle). This was caused by a change deployed at 09:50 to address an issue in the offline web app, which itself caused an issue with our identity and authentication service. On noticing this, we immediately took steps to rollback the change and restore service. We have taken an action to review and improve our pre- and post-deployment testing activities, including a review of our deployment environments. We are also temporarily suspending non-urgent deployments whilst this review takes place.

Posted: Tue, 23 Apr 2024 08:20:00 +0000

Resolved

Thank you for your patience whilst we worked on this. This issue has now been resolved and details of the cause will be shared on this page in due course.

Please accept our apologies for any inconvenience.

Posted: Mon, 22 Apr 2024 09:49:00 +0000

Investigating

We're currently investigating reports of disruption to service for learners. We'll keep this page updated with our progress.

Please accept our apologies for any inconvenience.

Posted: Mon, 22 Apr 2024 09:42:00 +0000

Degraded service: Kallidus Suite

» View Event Details | Created Mon, 22 Apr 2024 14:08:00 +0000

Post-Mortem

RCA: Between 11:40 and 16:00 BST on 22-04-2024, customers may have experienced slowness when accessing their Kallidus system. Following in-depth investigation , engineers discovered an increase time taken to process some routing requests which in some instances led to a further processing bottleneck. We have identified potential areas of improvement to these services and taken appropriate action. Since these changes were applied service has returned to normal. Ongoing monitoring suggests this solution has been successful and we continue to monitor closely. Please accept our apologies for any inconvenience caused by this disruption.

Posted: Thu, 25 Apr 2024 16:17:00 +0000

Resolved

Engineers have resolved issues relating to slowness or inability to access products in the Kallidus Suite. An RCA for this incident will be made available here in due course.

Apologies for the inconvenience caused

Posted: Tue, 23 Apr 2024 06:22:00 +0000

Update

Our engineers are continuing to diagnose the issue and are actively investigating. Please accept our apologies for the inconvenience this is causing today.

Posted: Mon, 22 Apr 2024 16:21:00 +0000

Update

Our engineers are aware of this issue and are actively investigating. The next update will be provided in 60 minutes, or sooner if the situation changes.

Posted: Mon, 22 Apr 2024 15:05:00 +0000

Investigating

Degraded service: Kallidus Suite Our engineers are currently investigating slowness to access login and pages within the Kallidus Suite. We will continue to update this page as our teams work to resolve the issues. Please accept our apologies for the inconvenience this is currently causing.

Posted: Mon, 22 Apr 2024 14:08:00 +0000

Degraded Service: Kallidus Learn

» View Event Details | Created Mon, 22 Apr 2024 17:05:00 +0000

Post-Mortem

RCA: Between 17:05 BST and 17:15 BST on 22/04/24 users would have been unable to access Learn administration. This was caused by an update to the User Education Panel that contained characters the application did not support. Once identified, the message was changed so that access could be restored. Although the issue was resolved at 17:15 BST, any users that were affected may have been unable to access administration beyond that time due to caching of the bad message. Cached messages were all cleared by 18:45 BST. We have identified an improvement that will handle these bad characters more gracefully, and whilst we implement that we will be more careful with the messages we post.

Posted: Wed, 24 Apr 2024 14:27:00 +0000

Resolved

Resolved: Access to Kallidus Learn administration functionality has been restored for affected customers.

Posted: Mon, 22 Apr 2024 17:15:00 +0000

Investigating

Access to Learn administration is currently restricted for a subset of Kallidus customers. We are working on restoring access as soon as possible.

Apologies for the inconvenience this is causing.

Posted: Mon, 22 Apr 2024 17:05:00 +0000

Degraded Service: Kallidus Sapling (US)

» View Event Details | Created Wed, 24 Apr 2024 10:25:00 +0000

Post-Mortem

RCA; Between the times of 19:15 on 23-04-2024 and 13:55 BST on 24-04-2024 customers may have intermittently experienced difficulties accessing some functionality within Kallidus HR. Preliminary root cause: Engineers identified an issue with Azure Load Balancers incorrectly routing traffic. Engineers removed the affected systems from the Azure Load Balancers and re-added them to clear the routing cache, restoring service.

Please accept our apologies for any inconvenience this may have caused and thank you for your patience while we worked to resolve this issue.

Posted: Fri, 26 Apr 2024 13:03:00 +0000

Resolved

Resolved: Our engineers have now resolved the issues being experienced by a sub-set of Kallidus Sapling (US) customers. An RCA for this incident will be posted in due course.

Posted: Wed, 24 Apr 2024 13:51:00 +0000

Update

Engineers are continuing to investigate the cause of the problem. There is no further update to share at this time.

Posted: Wed, 24 Apr 2024 11:44:00 +0000

Update

Engineers are continuing to investigate the cause of the problem. There is no further update to share at this time.

Posted: Wed, 24 Apr 2024 12:44:00 +0000

Investigating

Engineering are investigating reports of issues accessing some of the pages within Sapling, manifesting in an error being displayed, a blank screen or spinning wheel on the page being requested for a subset of customers. We will update here as soon as we discover more. Please accept our apologies for the inconvenience this is causing if you are currently affected.

Posted: Wed, 24 Apr 2024 10:25:00 +0000