Incident report 04/06/2024

Start: Karadoc/2.8.3 - 11:00am

End: Karadoc/2.8.4 - 1:55pm

Reach: limited to flows triggered by webhooks with payload mismatch errors (see details below)

Impact: high - blocking

Description

Shortly after the Karadoc release, we identified a regression bug affecting some webhook triggers. Our internal monitoring system was not properly activated to detect failure rates and alert our support team. Consequently, the issue was analyzed only after users reported problems with certain flows.

The impact was confined to workflows executed during the incident timeframe and triggered by HTTP requests where there was a mismatch between the content type set as URL-encoded and the actual content as a JSON payload. After the upgrade, it began failing, whereas it was previously accepted in a more lenient manner.

We sincerely apologize for any inconvenience this incident may have caused. We appreciate your patience and understanding as we work to prevent similar issues in the future.

Causes

• HTTP requests sent through webhooks with the application/x-www-form-urlencoded body type were incorrectly converted. This caused the webhook to have incoherent and unusable data, preventing steps from accessing the data.

Resolution

• The Mindflow engineering team patched the affected systems and fixed this webhook issue in Karadoc/2.8.4.

Follow-up

• The Mindflow development team will enhance the quality assurance suite for webhooks to cover this regression.
• The internal monitoring system will be reviewed and reactivated to detect issues promptly.