# Incident Report 13/08/2026

**Start:** `Raptor/3.20.13` (August 13, 2026, ~07:00 AM UTC+2)

**Identified:** August 13, 2026, ~10:00 AM UTC+2

**End:** Hotfix delivered August 13, 2026, ~02:20 PM UTC+2; remaining deployments corrected August 24, 2026

**Reach:** Deployments tracking the **latest** flow version, for flows that had been modified since the **Qilin** delivery

**Impact:** Medium — some executions could run an older flow snapshot after cutover; for deployments already pinned to a wrong snapshot and left unmodified afterward, that could continue until a forced redeploy

## Description

Since the **Qilin** delivery, Mindflow stores a **version snapshot** whenever a flow is modified. That allows a given execution to be inspected against the exact graph that was in effect at run time, instead of only against a single “latest” definition.

During the **Raptor/3.20.13** blue/green delivery, some deployments configured to follow the **latest** version could execute an **older** snapshot than expected.

The same-day hotfix fixed how Mindflow chooses the latest snapshot for **new** deploys. It did not automatically correct deployments that had already been resolved to a wrong snapshot between the Raptor cutover and the hotfix.

On **August 21, 2026**, we noticed that some of those **latest** deployments were still on the wrong snapshot when the flow had not been modified again after the incident window. As a precaution, on **August 24, 2026**, Mindflow triggered a progressive force redeploy of deployments tracking the latest version so those flows run the correct snapshot again.

## Cause

Latest snapshot selection relied on the object **last-modified** timestamp in object storage.

Blue/green delivery **re-uploads** version snapshot objects into the new environment. That rewrite sets a **shared write time** on those objects, so last-modified no longer reflects when each snapshot was actually created.

With several snapshots under the same version and identical last-modified times, selection of “latest” became **non-deterministic** and could resolve to an older snapshot. Only flows that had produced **multiple** snapshots since Qilin were exposed (unmodified flows still had a single snapshot).

## Impact

- Affected: flows modified since Qilin, with a deployment on **always on latest**, between the Raptor/3.20.13 cutover (~07:00 AM UTC+2) and the hotfix (~02:20 PM UTC+2).
- After the August 13 hotfix: new snapshot resolution was correct. Deployments that had already received the wrong snapshot during that window could keep running that older graph if the flow was not modified again afterward.
- Symptom: executions used an older flow graph.
- Unaffected: flows never modified since Qilin (single snapshot), and deployments pinned to a fixed version.

## Resolution

Snapshot ordering no longer uses storage last-modified time. Mindflow now orders nested snapshots by their **time-ordered snapshot identifier** (UUID version 7), which preserves creation order across blue/green copies.

The hotfix was delivered the same day (~02:20 PM UTC+2). That stopped new incorrect selections; it did not by itself rewrite deployments that already pointed at the wrong snapshot.

On **August 21, 2026**, we noticed remaining **latest** deployments still on a wrong snapshot from the cutover-to-hotfix window, when no later flow change had triggered a redeploy. On **August 24, 2026**, a progressive force redeploy was triggered for deployments tracking the latest flow version, to ensure those flows are running on the correct snapshot.

## Follow-up

- Automated tests cover blue/green-style identical timestamps and incorrect last-modified ordering, to prevent regressions.
- We will keep validating snapshot resolution as part of blue/green delivery checks.