» Published on Wed, 14 Mar 2018 00:00:00 +0000

• Post-Mortem

  Summary of impact:

  Between 04:07 and 11:44 UTC on 14-03-2018, all customers may have experienced difficulties with a subset of Perform application features: • 04:07-09:08 – some teams and members may not have been accessible through Team Drill Down • 04:07-09:09 – data added within this timeframe will not have been applied to statistics on the My Team, Hr Dashboard and View Everybody pages. Users added during this time period will not have been visible in search results. • 04:07-11:43 – reviews completed during this time will not be excluded from the ‘Reviews To Do’ statistic on the My Team, Hr Dashboard and View Everybody pages. Review Reassignments will not have been processed. • 04:07-11:44 - data added within this timeframe will not have been available through the data export

  All data is now present and correct as expected.

  Root cause and mitigation: Engineers determined that worker services stopped as a result of dropping their subscription to the EventStore. Engineers received numerous notifications from monitoring and following an investigation, they began work to resolve the issue at approximately 9am UTC by restarting the stopped services.

  Next steps: We sincerely apologize for the impact to affected customers. We are continuously taking steps to improve the Kallidus Suite Platform and our processes to help ensure such incidents do not occur in the future. In this case, this includes (but is not limited to): making the workers resilient to dropped subscriptions.

  » Updated Wed, 14 Mar 2018 12:49:00 +0000

• Resolved

  Summary of impact: Between 04:00 and 11:52 UTC on 14-03-2018, all customers may have experienced difficulties in using some aspects of Kallidus Suite applications. Impacted services include Perform.

  Preliminary root cause: Engineers determined that there was an issue with the EventStore subscription, impacting Kallidus Perform export data and outstanding review statistics.

  Mitigation: Engineers received numerous notifications from monitoring, and immediately remediated the issue by taking steps to reinitiate the process controlling the export data.

  Next steps: Engineers will continue to monitor and investigate to establish the detailed root cause, and the full root cause analysis post-mortem will be posted on this status page once completed.

  » Updated Wed, 14 Mar 2018 11:52:00 +0000

• Update

  Starting at 04:00 UTC on 14-03-2018 all Kallidus Perform customers may experience difficulties in using some aspects of Kallidus Suite applications. Impacted services include Perform. Engineers are actively working to resolve this issue. The next update will be provided in 60 minutes, or as events warrant.

  » Updated Wed, 14 Mar 2018 11:15:00 +0000

• Update

  Starting at 04:00 UTC on 14-03-2018 all Kallidus Perform customers may experience difficulties in using some aspects of Kallidus Suite applications. Impacted services include Perform. Engineers are aware of this issue and are actively investigating and working to resolve. The next update will be provided in 60 minutes, or as events warrant.

  » Updated Wed, 14 Mar 2018 10:17:00 +0000

• Investigating

  Engineers are investigating an alert on Kallidus Suite. This may affect Kallidus Perform export data and outstanding review statistics. Additional information will be provided shortly.

  » Updated Wed, 14 Mar 2018 09:17:41 +0000