» View Event Details | Created
Scheduled
Every Thursday 06:00 - 08:00 UK time we carry out essential maintenance including system updates and restarts. During this window, you may experience periods of degraded service.
» View Event Details | Created
Post-Mortem
Between 08:00 BST and 11:50 BST on 06-06-2024 a small percentage of customers were unable to access Learn or the administration area due to a startup error impacting users. Engineers worked to rollback the update that was causing this problem, and began rolling this out to affected customers from 10:30 BST. We will continue to investigate what caused the problem - as well as how and why it was not caught by our pre-release testing processes - to ensure the problem does not occur again. In the meantime engineers will look to implement the new functionality provided by this change in a different way. We are very sorry for the extended outage that some customers will have experienced.
Resolved
Our engineers have now resolved the access issues experienced by a sub-set of our Learn customers today. We would recommend closing out all browsers and signing back in to the product to ensure it's working as expected. Some customers may need to clear their cache/cookies before seeing the issue fully resolved.
An RCA for this incident will be posted here in due course.
Please accept our apologies once again for the inconvenience this caused today.
Update
Engineers are currently rolling out a fix for this and we hope to provide a positive message regarding a fix very shortly.
Thank you for your continued patience while we work through the issue to resolution and apologies again for the inconvenience this is causing.
Update
Engineers are continuing to investigate the cause of the issue, affecting a subset of our customers sites, following the maintenance window this morning and working on a rollback plan.
Thank you for your continued patience while we work through the issue to resolution and apologies again for the inconvenience this is causing.
Update
Engineers are continuing to investigate the cause of the issue, affecting a subset of our customers sites, following the maintenance window this morning.
Thank you for your continued patience while we work through the issue to resolution.
Investigating
We are aware of an issue for a subset of customers this morning accessing Learn, following our usual Thursday morning maintenance window. Our engineers are investigating and we will endevour to have the issue resolved as quickly as possible.
Apologies for any inconvenience this is causing.
» View Event Details | Created
Scheduled
Every Thursday 06:00 - 08:00 UK time we carry out essential maintenance including system updates and restarts. During this window, you may experience periods of degraded service.
» View Event Details | Created
Post-Mortem
Between 14:42 and 15:09 there was a severely degraded service within the Recruit application. This was due to extreme load a result of a number of local caches expiring at a similar time. Engineers tracked down the root cause to be an earlier restart of Recruit that synchronised the cache expiry times. This was an edge case event that is unlikely to occur again, however, we have updated our process to perform restarts at a point that will have less of an impact and we have added items to our backlog to increase the randomness of cache expiry to decrease the chances of multiple expiring at the same time.
Update
Our Engineers have resolved all Kallidus Recruit site issues in relation to the inability to access / login and all sites should be operational for customers.
An RCA for this incident will be posted here in due course.
Apologies once again if you were affected by this issue.
Investigating
We are aware of a subset of Kallidus Recruit customers site being inaccessible / unable to login. Our engineers are investigating and we will update this post as soon as we know more.
Apologies for any inconvenience this is causing currently.
» View Event Details | Created
Scheduled
Every Thursday 06:00 - 08:00 UK time we carry out essential maintenance including system updates and restarts. During this window, you may experience periods of degraded service.
» View Event Details | Created
Post-Mortem
Between 08:50 BST and 10:12 BST on 20/06/2024 customers would have been unable to launch Kallidus Off-the-shelf (OTS) content in Learn.
Following a software deployment, an issue with launching our OTS content was introduced to Kallidus Learn and will have displayed as an error: 'unable to process your request' to learners. This was linked to the unique way this type of content is hosted, and will not have impacted non-Kallidus OTS eLearning. While our standard post-deployment tests do include launching content, on this occasion this type was missed which led to a slight delay in identifying the issue and beginning remedial action.
In order to mitigate for the future, we have updated our automated testing processes to incorporate this type of eLearning content. We are also undertaking a review of our deployment and quality processes, to determine how this specific change made reached the release stage.
Please accept our apologies for any inconvenience caused.
Resolved
Resolved: Our engineers have now resolved the issue with access to our OTS courses from Learn sites now restored and available again.
Please accept our apologies for the inconvenience caused. An RCA for this incident will be posted here in due course.
Investigating
Our engineers are currently investigating an issue loading Off the Shelf Kallidus courses from our Learn environments this morning.
Please accept our apologies for any inconvenience this is causing.
» View Event Details | Created
Post-Mortem
At 15:48 UTC on 20th June 2024 a mis-formulated database cleanup script erroneously cleared bespoke customer settings and data fields from live tables within the US Kallidus-Sapling platform. As per policy, Kallidus maintains active backups of all production data and initiated the restore process. All missing fields were confirmed to be reinstated at or before 16:27 UTC on the 21st June 2024.
To ensure that there is no repetition of this incident, supplementary controls on routine database maintenance activities have been put in place which restrict how this activity is undertaken, alongside an additional level of scrutiny in the review of any scripts that may be ultimately deployed into the production environment.
Please accept our apologies for any inconvenience this incident caused.
Resolved
All data can now be viewed.
Details of the cause and mitigation steps will be shared in due course on this page.
Thank you for your patience whilst we resolved this issue, please accept our apologies for any inconvenience caused.
Update
Profile data is continuing to be restored and customers will start to see data re-appearing in the user profile page; There is still a sub-set of data to be restored back to pages but we will update as soon as this is fully completed. This issue continues to only affect a small sub-set of customers specifically in our N. America region, and we continue resolve as swiftly as possible. Thank you for your continued patience whilst we continue to work to resolve this.
Update
Profile data is now partially restored and customers will start to see data re-appearing in the user profile page from now; There is still a sub-set of data to be restored back to pages but we will update as soon as this is fully completed. This issue continues to only affect a small sub-set of customers specifically in our N. America region, and we continue resolve as swiftly as possible. Thank you for your continued patience whilst we continue to work to resolve this.
Update
Our engineers are in the final stages of testing a potential solution for the incident and we hope to provide some more positive news on our next update. This issue continues to only affect a small sub-set of customers specifically in our N. America region, and we continue resolve as swiftly as possible. Thank you for your continued patience whilst we continue to work to resolve this.
Update
Thank you for your continued patience whilst we work to resolve this issue affecting a subset of Sapling customers.
We will continue to update this page with our progress.
Update
Our engineers are continuing to work on the issue with profile data not displaying correctly. This issue continues to only affect a small sub-set of customers specifically in our N. America region, and we continue resolve as swiftly as possible. Thank you for your continued patience whilst we continue to work to resolve this.
Update
Our engineers are continuing to work on the issue with profile data not displaying correctly. This issue continues to only affect a small sub-set of customers specifically in our N. America region, and we continue resolve as swiftly as possible. Thank you for your continued patience whilst we continue to work to resolve this.
Update
Our engineers are continuing to work on the issue with profile data not displaying correctly.This issue continies to only affect a small sub-set of customers specifically in our N. America region, and we continue resolve as swiftly as possible. Thank you for your continued patience whilst we continue to work to resolve this.
Update
Some residual issues with data not displaying as expected on profiles is ongoing. This issue affects a small sub-set of customers specifically in the N.America region, and we continue resolve as swiftly as possible. Thank you for your continued patience whilst we work to resolve this.
Update
Some residual issues with data not displaying as expected on profiles is ongoing. This issue affects a small sub-set of customers specifically in the N.America region, and we continue resolve as swiftly as possible. Thank you for your continued patience whilst we work to resolve this.
Update
We are investigating reports of some residual issues with data not displaying as expected on profiles. This issue affects a small sub-set of customers, and we are working to resolve as swiftly as possible.
Thank you for your continued patience whilst we work to resolve this.
Resolved
All data is now visible as expected and normal service has resumed. You may now modify profile templates again if you wish.
Thank you for your patience whilst we worked on this, and please accept our apologies for any disruption this caused for you.
Details of the root cause will be shared on this page in due course.
Update
We are now in the process of restoring the ability to view data on profiles.
We will continue to update this page with our progress, thank you for your ongoing patience.
Update
Please accept our apologies for any inconvenience being caused by this issue.
We have identified the issue and are working to resolve it as swiftly as possible. We will continue to update this page with our progress.
Update
Thank you for your patience whilst we continue to work on resolving this issue.
We are making progress and will continue to share updates on this page.
Please accept our apologies for any inconvenience caused.
Update
Thank you for your patience whilst we continue to investigate. We have identified an issue linked to data not being displayed on some profile templates, all data is still present in the database.
We are working to resolve this issue as a priority now. In the meantime, please do not modify the profile templates. We will let you know once the issue is fixed, and you can then proceed with modifications.
We will continue to keep this page updated with our progress. Please accept our apologies for any inconvenience caused.
Investigating
We are investigating reports that some Sapling customers are experiencing issues accessing some pages, and some information is not presenting as normal.
We will keep this page updated with our investigation progress. Thank you for your patience whilst we look into this.
» View Event Details | Created
Scheduled
Every Thursday 06:00 - 08:00 UK time we carry out essential maintenance including system updates and restarts. During this window, you may experience periods of degraded service.