Write-up published
Resolved
This incident has been resolved. We will provide further information in a public incident report.
Summary
On Mar 12, 2025, from 09:13 to 12:03 UTC, most merchants were unable to login to their Admin and some shoppers experienced errors navigating or placing orders. This was triggered by a downstream dependency failure in the VTEX IO platform.
Our global sales flow was unaffected during this incident. Some accounts may have experienced sales below forecasted levels due to customized Checkout flows and payment connectors depending on VTEX IO infrastructure.
We apologize for any inconvenience this may have caused.
Timeline
At 09:13 UTC, our platform health monitoring indicated increased errors in VTEX IO.
At 09:17 UTC, our incident response team was notified of the issue.
At 11:43 UTC, we identified an issue in a downstream dependency used by a VTEX IO authZ service.
At 11:54 UTC, we performed a mitigation action and the platform started to gradually recover.
At 12:03 UTC, the incident was fully mitigated.
Monitoring
The fix for the issue in VTEX IO has been implemented.
Merchants should no longer be experiencing errors when attempting to login to their Admin. Shoppers should no longer be experiencing errors navigating or placing orders.
Our incident response team is monitoring to guarantee that normal platform behavior is fully reestablished.
We will send an additional update in the next 30 minutes, or as soon as we have more information to share.
Identified
We have identified the contributing factors of the issue in VTEX IO and implemented a fix.
Our incident response team estimates that gradual recovery will be observed within the next few minutes.
We will send an additional update in the next 30 minutes, or as soon as we have more information to share.
Investigating
We continue investigating the issue in VTEX IO.
We have attempted different mitigation actions, but they were not effective in reducing errors.
Our incident response team is working to identify the root cause and implement a solution.
We will send an additional update in the next 30 minutes, or as soon as we have more information to share.
Investigating
We continue investigating the issue in VTEX IO.
Besides previously reported symptoms, additional impact may be seen in customized Checkout flows and payment connectors depending on VTEX IO infrastructure.
Our incident response team is working to identify the root cause and implement a solution.
We will send an additional update in the next 30 minutes, or as soon as we have more information to share.
Investigating
We continue investigating the issue in VTEX IO.
Merchants may be experiencing "Something went wrong" errors when attempting to login to their Admin. Shoppers may be experiencing "The circuit is now open and is not allowing calls" errors or infinite loading in some customized storefront components.
Our incident response team is working to identify the root cause and implement a solution.
We will send an additional update in the next 30 minutes, or as soon as we have more information to share.
Investigating
We are currently investigating an issue in a VTEX IO cluster.
Some merchants may be experiencing difficulties logging in to their Admin. Other unexpected errors may occur in customized apps used in storefronts.
Our incident response team is working to identify the root cause and implement a solution.
We will send an additional update in the next 30 minutes, or as soon as we have more information to share.
