Write-up published
Read our write-up of this incident at https://vtex-incident-reports.s3.us-east-1.amazonaws.com/public/2025-08-26_Public-Incident-Report_3743.pdf
Resolved
This incident has been resolved. We will provide further information in a public incident report.
Summary
On Aug 26, 2025, from 19:44 to 20:10 UTC, most shoppers experienced issues placing orders.
This incident was triggered by an error introduced in a recent change in a shared IAM policy used by our services. We mitigated its impact by reverting the shared IAM policy to its previous version.
Our sales flow was affected during this incident, impacting all accounts for 26 minutes.
We apologize for any inconvenience this may have caused.
Timeline
At 19:44 UTC, an error was introduced in a shared IAM policy used by our services.
At 19:46 UTC, our incident response team was notified of a sharp drop in orders in our platform.
At 20:09 UTC, we identified and reverted the recent change in the shared IAM policy.
At 20:10 UTC, the incident was fully mitigated.
Monitoring
The fix for the issue in Checkout has been implemented.
Shoppers should no longer be experiencing issues placing orders.
Our incident response team is monitoring to guarantee that normal platform behavior is fully reestablished.
We will send an additional update in the next 30 minutes, or as soon as we have more information to share.
Identified
We have identified the contributing factors of the issue in Master Data.
Our incident response team estimates that the fix will be implemented in the next 10 minutes.
We will send an additional update in the next 30 minutes, or as soon as we have more information to share.
Investigating
We are currently investigating an issue in Master Data, affecting Checkout.
Shoppers may be experiencing issues placing orders.
Our incident response team is working to identify the root cause and implement a solution.
We will send an additional update in the next 30 minutes, or as soon as we have more information to share.
