Incident report 09/01/2025

Start: Rafiki/2.15.24 - 18:30 (Europe/Paris)

End: Sauron/2.16.10 - 21:30 (Europe/Paris)

Reach: all flows (see details below)

Impact: high - blocking

Description

While releasing Sauron, bugs were identified that impacted the execution of workflows. The issue caused a temporary denial of some playbook executions for approximately 3 hours during the incident timeframe.

Analytics are currently not functional due to a change in the format of the data in the database.

In the studio, executions triggered by webhooks and email hooks will appear mixed in the logs pane.

We sincerely apologize for any inconvenience this may have caused and appreciate your patience as we work to prevent similar issues in the future.

Causes

• An internal API used to dispatch configuration across the codebase was updated without backward compatibility. This caused API consumers to fail for approximately 15 minutes (from 18:30 to 18:45), impacting workflows that included service steps. Service was fully restored after the codebase update was completed.
• A database migration failed silently, causing delays in subsequent migrations. Some AI-generated playbooks were in an invalid state, leading to the migration tool’s failure. The faulty playbooks were fixed, and the migration was successfully completed without further issues.
• The introduction of the new trigger steps required the migration of playbook hooks configuration. During this process, an issue was identified in the database attributes: older playbooks (created before the introduction of the enable/disable hooks feature) and newer playbooks had mismatched attributes, causing the migration to fail. Using playbook execution logs, we identified the affected playbooks that were run via email hooks and webhooks. Their hooks were reactivated accordingly, and the migration was re-executed successfully. This issue resulted in the denial of playbook executions for approximately 3 hours (from 18:30 to 21:30). Service was manually restored, and operations have since returned to normal.

We apologize for any inconvenience caused and are implementing measures to avoid such issues in the future.

Resolution

• The first issue was resolved upon completing the Sauron release deployment.
• The second issue was addressed by updating the attributes of AI-generated playbooks.
• The last issue was resolved through a manual investigation to identify the source of the playbook executions.

Follow-up

• The Mindflow development team will strengthen the quality assurance process for internal API changes.
• Mindflow will focus on recovering access to analytics and specific logs in the studio to clearly differentiate email hook runs from webhook runs.