Incident report 09/01/2024

Start: Zoza/1.14.2 - 3:40pm

End: Azeroth/1.15.2 - 5:15pm

Reach: limited to flows using Adaptive Shield integration (see below)

Impact: high - blocking

Description

During the rollout of Azeroth release, we encountered a bug affecting certain playbooks using Adaptive Shield.

On May 17, 2023, Adaptive Shield was introduced as a custom integration in a few select customer and internal environments. One of the updates of Azeroth release was to clean up and migrate the custom integration to make it available to all our customers. Unfortunately, the migration failed, causing disruptions in the usage of this integration.

This document briefly details the cause, resolution, and subsequent actions to be taken. The Mindflow team apologizes for the inconvenience and extends thanks to our client’s team for their support and understanding.

Causes

Three issues were identified during this incident:

1. Integrations were not updated in our production environment, only in staging.
2. A planned migration failed because Adaptive Shield credentials were not migrated properly. This issue occurred only when an Adaptive Shield step was executed.
3. Due to the nature of the migration, the running workflows were not properly updated despite automatic deployment.

Upon receiving the report, the Mindflow team promptly identified and analyzed those issues.

Resolution

The Mindflow engineering team resolved the issue with assistance from customers, addressing the problems affecting some playbooks by:

• Properly updating integrations in the production environment.
• Manually migrating credentials.
• Manually forcing flows using the integration to redeploy.

Follow-up

• The Mindflow development team will enhance and automate their release process for the delivery of integrations.
• Adaptive Shield integration is no longer considered a legacy and will be included in platform updates as necessary.