Incident Report 13/08/2026
Start: Raptor/3.20.13 (August 13, 2026, ~07:00 AM UTC+2)
Identified: August 13, 2026, ~10:00 AM UTC+2
End: Hotfix delivered August 13, 2026, ~02:20 PM UTC+2; remaining deployments corrected August 24, 2026
Reach: Deployments tracking the latest flow version, for flows that had been modified since the Qilin delivery
Impact: Medium — some executions could run an older flow snapshot after cutover; for deployments already pinned to a wrong snapshot and left unmodified afterward, that could continue until a forced redeploy
Description
Section titled “Description”Since the Qilin delivery, Mindflow stores a version snapshot whenever a flow is modified. That allows a given execution to be inspected against the exact graph that was in effect at run time, instead of only against a single “latest” definition.
During the Raptor/3.20.13 blue/green delivery, some deployments configured to follow the latest version could execute an older snapshot than expected.
The same-day hotfix fixed how Mindflow chooses the latest snapshot for new deploys. It did not automatically correct deployments that had already been resolved to a wrong snapshot between the Raptor cutover and the hotfix.
On August 21, 2026, we noticed that some of those latest deployments were still on the wrong snapshot when the flow had not been modified again after the incident window. As a precaution, on August 24, 2026, Mindflow triggered a progressive force redeploy of deployments tracking the latest version so those flows run the correct snapshot again.
Latest snapshot selection relied on the object last-modified timestamp in object storage.
Blue/green delivery re-uploads version snapshot objects into the new environment. That rewrite sets a shared write time on those objects, so last-modified no longer reflects when each snapshot was actually created.
With several snapshots under the same version and identical last-modified times, selection of “latest” became non-deterministic and could resolve to an older snapshot. Only flows that had produced multiple snapshots since Qilin were exposed (unmodified flows still had a single snapshot).
Impact
Section titled “Impact”- Affected: flows modified since Qilin, with a deployment on always on latest, between the Raptor/3.20.13 cutover (~07:00 AM UTC+2) and the hotfix (~02:20 PM UTC+2).
- After the August 13 hotfix: new snapshot resolution was correct. Deployments that had already received the wrong snapshot during that window could keep running that older graph if the flow was not modified again afterward.
- Symptom: executions used an older flow graph.
- Unaffected: flows never modified since Qilin (single snapshot), and deployments pinned to a fixed version.
Resolution
Section titled “Resolution”Snapshot ordering no longer uses storage last-modified time. Mindflow now orders nested snapshots by their time-ordered snapshot identifier (UUID version 7), which preserves creation order across blue/green copies.
The hotfix was delivered the same day (~02:20 PM UTC+2). That stopped new incorrect selections; it did not by itself rewrite deployments that already pointed at the wrong snapshot.
On August 21, 2026, we noticed remaining latest deployments still on a wrong snapshot from the cutover-to-hotfix window, when no later flow change had triggered a redeploy. On August 24, 2026, a progressive force redeploy was triggered for deployments tracking the latest flow version, to ensure those flows are running on the correct snapshot.
Follow-up
Section titled “Follow-up”- Automated tests cover blue/green-style identical timestamps and incorrect last-modified ordering, to prevent regressions.
- We will keep validating snapshot resolution as part of blue/green delivery checks.