This incident has been resolved. We will not provide further information in a public incident report.

Summary

On Aug 28, 2025 from 19:45 to 21:15 UTC, developers experienced issues seeing FastStore deployments in the Admin. During this period, most deployments failed, meaning it was not possible to make changes to published storefronts. However, no impact in production stores was observed throughout this incident.

This incident was triggered by a CI/CD bug in FastStore WebOps, which pointed production traffic to a service version that did not properly handle recent changes in our database structure. We mitigated its impact by reverting FastStore WebOps to a version that was ready to handle those changes.

Our sales flow was unaffected during this incident, but storefront updates for a few accounts using FastStore WebOps was impacted for 1 hour and 30 minutes.

We apologize for any inconvenience this may have caused.

Timeline

At 19:45 UTC, a CI/CD bug in FastStore WebOps pointed production traffic to a faulty version of the service.

At 20:30 UTC, we started receiving customer reports of issues seeing FastStore deployments in the Admin.

At 21:03 UTC, our incident response team was notified of the issue.

At 21:14 UTC, reverted production traffic to a healthy version of the service.

At 21:15 UTC, the incident was fully mitigated.

Recheduled for Sep 4, 2025 at 06:00 - 07:00 UTC

We will work on improvements to our platform that require controlled interruptions in VTEX IO.

As we upgrade our VTEX IO databases, some stores might experience brief periods (under 20 minutes) of interruption in Admin operations involving apps, including app and workspace updates. Developers may experience issues linking and release new versions of apps. Storefront and Admin apps that use VBase for storage may experience temporary disruptions.

This is required to increase system reliability during periods of peak traffic and sales. We thank you in advance for your understanding as we upgrade our platform to serve you better, our valued customers and partners.

Maintenance has completed

This incident has been resolved. We will not provide further information in a public incident report.

Summary

On Aug 27, 2025 from 11:13 to 13:00 UTC, shoppers experienced delays when attempting to login by access code and receiving transactional emails. Merchants experienced issues receiving reports by email.

This incident was triggered by a malicious actor, who performed an attack targeted at specific store who had a public contact form, which was abused through automated bots. We mitigated its impact by identifying the source of the sharp increase in mail service requests and blocking it.

The impact recurred from 13:27 to 14:06 UTC because we were still processing a large backlog of messages and under a restricted cloud provider quota for mail delivery.

Our sales flow was not significantly affected during this incident, but mail delivery was severely impacted for most accounts for 2 hours and 26 minutes across two impact windows.

We apologize for any inconvenience this may have caused.

Timeline

At 11:17 UTC, our incident response team was notified of the issue.

At 11:45 UTC, our team identified there was a sharp increase in mail service requests at 11:13 UTC.

At 12:26 UTC, our team identified the source of the sharp increase in mail service requests.

At 13:00 UTC, our team blocked the abuser and the incident was fully mitigated.

At 13:27 UTC, our team received new reports of mail delivery issues.

At 13:28 UTC, our team identified that we were under a restricted cloud provider quota for mail delivery.

At 14:06 UTC, our cloud provider approved a temporary quota increase for our account, mitigating impact on mail delivery once again. We continued monitoring until 15:28 UTC, when a larger quota increase ensured continued platform stability for mail delivery until the large backlog of messages in our queue is processed.

Maintenance has completed

This incident has been resolved. We will provide further information in a public incident report.

Summary

On Aug 26, 2025, from 19:44 to 20:10 UTC, most shoppers experienced issues placing orders.

This incident was triggered by an error introduced in a recent change in a shared IAM policy used by our services. We mitigated its impact by reverting the shared IAM policy to its previous version.

Our sales flow was affected during this incident, impacting all accounts for 26 minutes.

We apologize for any inconvenience this may have caused.

Timeline

At 19:44 UTC, an error was introduced in a shared IAM policy used by our services.

At 19:46 UTC, our incident response team was notified of a sharp drop in orders in our platform.

At 20:09 UTC, we identified and reverted the recent change in the shared IAM policy.

At 20:10 UTC, the incident was fully mitigated.

Maintenance has completed

Maintenance has completed

This incident has been resolved. We will not provide further information in a public incident report.

Summary

On Aug 7, 2025 from 18:33 to 18:46 UTC, some shoppers experienced issues placing orders and increased 5xx errors in Checkout simulation flows. This was triggered by a bug that was introduced in a recent deployment to our Master Data service, which caused a disruption to our Logistics shipping service.

Our global sales flow was partially affected for 13 minutes during this incident.

We apologize for any inconvenience this may have caused.

Timeline

At 18:13 UTC, a bug that was introduced in a recent deployment to our Master Data service.

At 18:33 UTC, our platform health monitoring displays an increase in 5xx errors in our Logistics shipping service.

At 18:37 UTC, our incident response team was notified of the issue.

At 18:42 UTC, our team identified the faulty deployment and initiated rollback procedures.

At 18:46 UTC, the incident was fully mitigated.

Maintenance has completed

This incident has been resolved. We will provide further information in a public incident report.

Summary

On Aug 5, 2025 from 12:33 to 12:50 UTC, merchants and shoppers faced issues logging in and placing orders. This was an unintended side-effect of an infrastructure configuration change in VTEX ID.

Our global sales flow was partially affected for 17 minutes during this incident.

We apologize for any inconvenience this may have caused.

Timeline

At 12:30 UTC, an infrastructure configuration change was completed by our team.

At 12:33 UTC, our platform health monitoring indicates that orders dropped below forecasted levels for the period.

At 12:36 UTC, our incident response team was notified of the issue.

At 12:47 UTC, our incident response team identified that the infrastructure configuration change was the likely trigger for the drop in orders and started reverting the change.

At 12:50 UTC, the infrastructure configuration change was completely reverted and normal platform behavior was reestablished.

Maintenance has completed

This incident has been resolved. We will provide further information in a public incident report.

Summary

On Aug 4, 2025 from 14:41 to 15:06 UTC, merchants and shoppers faced issues logging in and placing orders. This was an unintended side-effect of an infrastructure configuration change in VTEX ID.

Our global sales flow was partially affected for 25 minutes during this incident.

We apologize for any inconvenience this may have caused.

Timeline

At 14:38 UTC, an infrastructure configuration change was completed by our team.

At 14:41 UTC, our platform health monitoring indicates that orders dropped below forecasted levels for the period.

At 14:43 UTC, our incident response team was notified of the issue.

At 14:51 UTC, our incident response team identified that the infrastructure configuration change was the likely trigger for the drop in orders and started reverting the change.

At 15:06 UTC, the infrastructure configuration change was completely reverted and normal platform behavior was reestablished.

The maintenance has been completed.