This incident has been resolved. We will not provide further information in a public incident report.
Summary
On Aug 28, 2025 from 19:45 to 21:15 UTC, developers experienced issues seeing FastStore deployments in the Admin. During this period, most deployments failed, meaning it was not possible to make changes to published storefronts. However, no impact in production stores was observed throughout this incident.
This incident was triggered by a CI/CD bug in FastStore WebOps, which pointed production traffic to a service version that did not properly handle recent changes in our database structure. We mitigated its impact by reverting FastStore WebOps to a version that was ready to handle those changes.
Our sales flow was unaffected during this incident, but storefront updates for a few accounts using FastStore WebOps was impacted for 1 hour and 30 minutes.
We apologize for any inconvenience this may have caused.
Timeline
At 19:45 UTC, a CI/CD bug in FastStore WebOps pointed production traffic to a faulty version of the service.
At 20:30 UTC, we started receiving customer reports of issues seeing FastStore deployments in the Admin.
At 21:03 UTC, our incident response team was notified of the issue.
At 21:14 UTC, reverted production traffic to a healthy version of the service.
At 21:15 UTC, the incident was fully mitigated.
Recheduled for Sep 4, 2025 at 06:00 - 07:00 UTC
We will work on improvements to our platform that require controlled interruptions in VTEX IO.
As we upgrade our VTEX IO databases, some stores might experience brief periods (under 20 minutes) of interruption in Admin operations involving apps, including app and workspace updates. Developers may experience issues linking and release new versions of apps. Storefront and Admin apps that use VBase for storage may experience temporary disruptions.
This is required to increase system reliability during periods of peak traffic and sales. We thank you in advance for your understanding as we upgrade our platform to serve you better, our valued customers and partners.
Maintenance has completed
This incident has been resolved. We will not provide further information in a public incident report.
Summary
On Aug 27, 2025 from 11:13 to 13:00 UTC, shoppers experienced delays when attempting to login by access code and receiving transactional emails. Merchants experienced issues receiving reports by email.
This incident was triggered by a malicious actor, who performed an attack targeted at specific store who had a public contact form, which was abused through automated bots. We mitigated its impact by identifying the source of the sharp increase in mail service requests and blocking it.
The impact recurred from 13:27 to 14:06 UTC because we were still processing a large backlog of messages and under a restricted cloud provider quota for mail delivery.
Our sales flow was not significantly affected during this incident, but mail delivery was severely impacted for most accounts for 2 hours and 26 minutes across two impact windows.
We apologize for any inconvenience this may have caused.
Timeline
At 11:17 UTC, our incident response team was notified of the issue.
At 11:45 UTC, our team identified there was a sharp increase in mail service requests at 11:13 UTC.
At 12:26 UTC, our team identified the source of the sharp increase in mail service requests.
At 13:00 UTC, our team blocked the abuser and the incident was fully mitigated.
At 13:27 UTC, our team received new reports of mail delivery issues.
At 13:28 UTC, our team identified that we were under a restricted cloud provider quota for mail delivery.
At 14:06 UTC, our cloud provider approved a temporary quota increase for our account, mitigating impact on mail delivery once again. We continued monitoring until 15:28 UTC, when a larger quota increase ensured continued platform stability for mail delivery until the large backlog of messages in our queue is processed.
Maintenance has completed
This incident has been resolved. We will provide further information in a public incident report.
Summary
On Aug 5, 2025 from 12:33 to 12:50 UTC, merchants and shoppers faced issues logging in and placing orders. This was an unintended side-effect of an infrastructure configuration change in VTEX ID.
Our global sales flow was partially affected for 17 minutes during this incident.
We apologize for any inconvenience this may have caused.
Timeline
At 12:30 UTC, an infrastructure configuration change was completed by our team.
At 12:33 UTC, our platform health monitoring indicates that orders dropped below forecasted levels for the period.
At 12:36 UTC, our incident response team was notified of the issue.
At 12:47 UTC, our incident response team identified that the infrastructure configuration change was the likely trigger for the drop in orders and started reverting the change.
At 12:50 UTC, the infrastructure configuration change was completely reverted and normal platform behavior was reestablished.
Maintenance has completed