This incident has been resolved. We will not provide further information in a public incident report.
Summary
On Aug 28, 2025 from 19:45 to 21:15 UTC, developers experienced issues seeing FastStore deployments in the Admin. During this period, most deployments failed, meaning it was not possible to make changes to published storefronts. However, no impact in production stores was observed throughout this incident.
This incident was triggered by a CI/CD bug in FastStore WebOps, which pointed production traffic to a service version that did not properly handle recent changes in our database structure. We mitigated its impact by reverting FastStore WebOps to a version that was ready to handle those changes.
Our sales flow was unaffected during this incident, but storefront updates for a few accounts using FastStore WebOps was impacted for 1 hour and 30 minutes.
We apologize for any inconvenience this may have caused.
Timeline
At 19:45 UTC, a CI/CD bug in FastStore WebOps pointed production traffic to a faulty version of the service.
At 20:30 UTC, we started receiving customer reports of issues seeing FastStore deployments in the Admin.
At 21:03 UTC, our incident response team was notified of the issue.
At 21:14 UTC, reverted production traffic to a healthy version of the service.
At 21:15 UTC, the incident was fully mitigated.
Resolved
This incident has been resolved. We will not provide further information in a public incident report.
Summary
On Aug 28, 2025 from 19:45 to 21:15 UTC, developers experienced issues seeing FastStore deployments in the Admin. During this period, most deployments failed, meaning it was not possible to make changes to published storefronts. However, no impact in production stores was observed throughout this incident.
This incident was triggered by a CI/CD bug in FastStore WebOps, which pointed production traffic to a service version that did not properly handle recent changes in our database structure. We mitigated its impact by reverting FastStore WebOps to a version that was ready to handle those changes.
Our sales flow was unaffected during this incident, but storefront updates for a few accounts using FastStore WebOps was impacted for 1 hour and 30 minutes.
We apologize for any inconvenience this may have caused.
Timeline
At 19:45 UTC, a CI/CD bug in FastStore WebOps pointed production traffic to a faulty version of the service.
At 20:30 UTC, we started receiving customer reports of issues seeing FastStore deployments in the Admin.
At 21:03 UTC, our incident response team was notified of the issue.
At 21:14 UTC, reverted production traffic to a healthy version of the service.
At 21:15 UTC, the incident was fully mitigated.
Monitoring
The fix for the issue in FastStore WebOps has been implemented.
Developers should no longer be experiencing issues seeing FastStore deployments in Admin. No impact in production stores was observed throughout this incident.
Our incident response team is monitoring to guarantee that normal platform behavior is fully reestablished.
We will send an additional update in the next 30 minutes, or as soon as we have more information to share.
Investigating
We are currently investigating an issue in FastStore WebOps.
Developers may be experiencing issues seeing FastStore deployments in Admin. No impact in production stores is observed.
Our incident response team is working to identify the root cause and implement a solution.
We will send an additional update in the next 30 minutes, or as soon as we have more information to share.
