» Published on Fri, 27 Apr 2018 00:00:00 +0000

• Post-Mortem

  Between 14:30 on 26.04.18 and 09:25 BST on 01.05.18 all customers may have experienced difficulties with the Excel export function in Kallidus Perform, where data within this time frame wasn’t available in the exported data.

  Root cause and mitigation: Engineers determined that a disk error had occurred, causing minor data corruption that impacted the Excel export function in Perform. No other functionality was affected and the application remained available at all times. There was no data loss from this incident. Engineers received numerous notifications from monitoring and remediated the issue by rebuilding the corrupt data from the redundant system.

  Next steps: We sincerely apologize for the impact to affected customers. We are continuously taking steps to improve the Kallidus Suite Platform and our processes to help ensure such incidents do not occur in the future. In this case, this includes (but is not limited to):

  • Replacing the disk subsystem with SSD technology which is more performant and resilient.
  • Working with the database supplier to understand the cause and impact in more detail so that monitoring can be improved.
  » Updated Fri, 04 May 2018 10:21:00 +0000

• Resolved

  Summary of impact: Between 14:30 on 26-04-2018 and 09:30 UTC on 01-05-2018, all customers may have experienced some functional degradation. Impacted services include Perform.

  Preliminary root cause: Engineers determined that a data corruption had occurred, impacting export data updates.

  Mitigation: Engineers received numerous notifications from monitoring, and immediately remediated the issue by restoring the impacted server node on the cluster.

  Next steps: Engineers will continue to monitor and investigate to establish the detailed root cause, and the full root cause analysis post-mortem will be posted on this status page once completed. Next steps: Engineers will continue to monitor and investigate to establish the detailed root cause, and the full root cause analysis post-mortem will be posted on this status page once completed.

  » Updated Tue, 01 May 2018 08:26:00 +0000

• Update

  Engineers are working to resolve this issue affecting exports on Kallidus Perform. Exports will not contain any data changes that took place after 2:30pm (BST) 26.04.18, but can still be run and will show data from before this time. Engineers are working to investigate and resolve this issue. The next update will be provided in 60 minutes, or as events warrant.

  » Updated Mon, 30 Apr 2018 08:16:00 +0000

• Update

  Starting at 14:30 BST on 26.04.18 all Kallidus Perform customers will experience issues with viewing data in exports from after this time. All data entered into the system before this time is available in exports. Engineers are aware of this issue and are actively investigating. The next update will be provided in 60 minutes, or as events warrant.

  » Updated Fri, 27 Apr 2018 11:16:00 +0000

• Investigating

  Engineers are investigating an alert on Kallidus Suite. This is affecting exports on Kallidus Perform. Exports will not contain any data changes that took place after 2:30pm (BST) yesterday (26.04.18), but can still be run and will show data from before this time. Engineers are working to investigate and resolve this issue. The next update will be provided in 60 minutes, or as events warrant.

  » Updated Fri, 27 Apr 2018 09:06:13 +0000