Incident report 01/10/2024

Start: Popeye/2.13.4 - 6:30am

End: Quasimodo/2.14.1 - 6:45am

Reach: all flows (see details below)

Impact: high - blocking

Description

During the Quasimodo release deployment, a bug was identified due to an internal API migration related to credentials, impacting some workflows. The issue affected workflows executed during the incident timeframe.

We sincerely apologize for any inconvenience this may have caused and appreciate your patience as we take steps to prevent similar issues in the future.

Causes

• The issue arose due to a change in how our system handles the legacy Authorized URLs parameter. Previously, our API returned this parameter, which the workflow engine depended on. In the Neo release, a new mechanism was introduced, deprecating Authorized URLs, though it was still retained. However, in the Quasimodo release, we fully removed this parameter from the API.
• The problem occurred because the API was updated before the workflow engine. As a result, the API stopped providing Authorized URLs while the workflow engine still expected it, causing failures in affected workflows.

Resolution

• The issue was resolved after the Quasimodo release deployment was completed.

Follow-up

• The Mindflow development team will strengthen the quality assurance process for internal API changes.