Incident Report 12/11/2025

Start: Fenrir/3.8.17(Nov 01, 2025, 12:55 AM UTC+2:00)

End: Fenrir/3.8.17 (Nov 04, 2025)

Reach: For certain environments, flow deployments for flows with the latest version that were modified during this time window

Impact: High – blocking

---

Description

Without any manual changes, a working flow suddenly stopped functioning with the following error message:

Failed to execute flow → The execution failed so the flow was stopped

---

Causes

To help identify the root cause if it occurs again, we have implemented additional logging and monitoring.

Here are technical details about the underlying cause:

• When you work on a flow—adding steps, changing settings, etc.—the flow object is saved in a database. When a modified flow is executed, a snapshot is created and saved as a file. The flow deployment process uses this file to run the flow.
• Under certain conditions, the latest version of this file was removed from our file storage, making execution impossible. However, the flow remains in the database, and all previous versions remain in our versioned file storage.
• A lifecycle policy introduced with recent infrastructure improvements caused this issue. This policy triggered the deletion of the internal snapshot (no data was lost) from file storage during the Fenrir deployment.

---

Resolution

As a temporary workaround, making a minor modification to the affected flow—such as moving a step and run manually the flow—triggers a redeployment and restores normal functionality.

The team is actively creating a hotfix to restore impacted flow deployments automatically. This report will be updated as new information becomes available.

---

Follow-up

The issue will not occur with the upcoming release for unchanged flow deployments. We sincerely apologize for this disruption. We understand the inconvenience this has caused and are committed to preventing similar issues in the future.