History

August
28
Thu
FastStore WebOps not displaying deployments in admin
9:44 PM

This incident has been resolved. We will not provide further information in a public incident report.

Summary

On Aug 28, 2025 from 19:45 to 21:15 UTC, developers experienced issues seeing FastStore deployments in the Admin. During this period, most deployments failed, meaning it was not possible to make changes to published storefronts. However, no impact in production stores was observed throughout this incident.

This incident was triggered by a CI/CD bug in FastStore WebOps, which pointed production traffic to a service version that did not properly handle recent changes in our database structure. We mitigated its impact by reverting FastStore WebOps to a version that was ready to handle those changes.

Our sales flow was unaffected during this incident, but storefront updates for a few accounts using FastStore WebOps was impacted for 1 hour and 30 minutes.

We apologize for any inconvenience this may have caused.

Timeline

At 19:45 UTC, a CI/CD bug in FastStore WebOps pointed production traffic to a faulty version of the service.

At 20:30 UTC, we started receiving customer reports of issues seeing FastStore deployments in the Admin.

At 21:03 UTC, our incident response team was notified of the issue.

At 21:14 UTC, reverted production traffic to a healthy version of the service.

At 21:15 UTC, the incident was fully mitigated.

VTEX IO database maintenance
9:36 PM

Recheduled for Sep 4, 2025 at 06:00 - 07:00 UTC

We will work on improvements to our platform that require controlled interruptions in VTEX IO.

As we upgrade our VTEX IO databases, some stores might experience brief periods (under 20 minutes) of interruption in Admin operations involving apps, including app and workspace updates. Developers may experience issues linking and release new versions of apps. Storefront and Admin apps that use VBase for storage may experience temporary disruptions.

This is required to increase system reliability during periods of peak traffic and sales. We thank you in advance for your understanding as we upgrade our platform to serve you better, our valued customers and partners.

VTEX IO database maintenance
7:00 AM

Maintenance has completed

27
Wed
Transactional email delivery issues
3:38 PM

This incident has been resolved. We will not provide further information in a public incident report.

Summary

On Aug 27, 2025 from 11:13 to 13:00 UTC, shoppers experienced delays when attempting to login by access code and receiving transactional emails. Merchants experienced issues receiving reports by email.

This incident was triggered by a malicious actor, who performed an attack targeted at specific store who had a public contact form, which was abused through automated bots. We mitigated its impact by identifying the source of the sharp increase in mail service requests and blocking it.

The impact recurred from 13:27 to 14:06 UTC because we were still processing a large backlog of messages and under a restricted cloud provider quota for mail delivery.

Our sales flow was not significantly affected during this incident, but mail delivery was severely impacted for most accounts for 2 hours and 26 minutes across two impact windows.

We apologize for any inconvenience this may have caused.

Timeline

At 11:17 UTC, our incident response team was notified of the issue.

At 11:45 UTC, our team identified there was a sharp increase in mail service requests at 11:13 UTC.

At 12:26 UTC, our team identified the source of the sharp increase in mail service requests.

At 13:00 UTC, our team blocked the abuser and the incident was fully mitigated.

At 13:27 UTC, our team received new reports of mail delivery issues.

At 13:28 UTC, our team identified that we were under a restricted cloud provider quota for mail delivery.

At 14:06 UTC, our cloud provider approved a temporary quota increase for our account, mitigating impact on mail delivery once again. We continued monitoring until 15:28 UTC, when a larger quota increase ensured continued platform stability for mail delivery until the large backlog of messages in our queue is processed.

Marketplace Bridge database maintenance
3:00 AM

Maintenance has completed

June
No incidents reported