Outage on FastStore stores
Incident Report for VTEX

From 15h30 UTC to 16h10 UTC, FastStore WebOps faced an outage caused by its Router service. FastStore WebOps is the underlying platform where clients using FastStore deploy their stores.

At 15h40 UTC, we identified that our router service had been deleted, and we promptly started the restore process. Meanwhile, we opened a separate investigation to map what happened. At 15h48 UTC, we finished the main cluster mitigation and restored other support infrastructure. 

At 15h56 UTC, FastStore traffic was restored, but a secondary service used to serve optimized images was still unreachable. At 16h10 UTC, our image optimizer service was restored. 

For us to continue to be the trusted partner to the success of our clients, our team is working on follow-up actions to ensure that this incident does not happen again and that we identify and recover from future incidents faster. We are committed to improving our systems to guarantee a reliable and trusted platform.

Read the complete report here.

Posted Feb 07, 2023 - 10:11 GMT-03:00

At 3:30 PM (GMT), our team identified an issue in our FastStore WebOps, the underlying infrastructure of stores built on FastStore. This instability affected all FastStore applications. Our team mitigated the problem at 3:56 PM (GMT), and we had stores backing to normal right after the mitigation action.

Our sincere apologies for the disruption in our services today. We appreciate you bearing with us in the meantime. Once we finish a detailed public report, we will update this status with the link.
Posted Feb 06, 2023 - 14:27 GMT-03:00
The issue with FastStore WebOps is solved. All stores are fully operational. We are monitoring our systems to ensure that we fully resolved the issue.
Posted Feb 06, 2023 - 13:25 GMT-03:00
We identified the issue and promptly fixed it. Stores running on FastStore WebOps are back to normal. But, we are still experiencing partial disruption in loading assets from the FastStore application.
Posted Feb 06, 2023 - 13:09 GMT-03:00
We’re currently investigating an outage in FastStore WebOps, and we’ll keep you posted as soon as we have more information. All other Storefronts(Portal CMS, VTEX IO Render) are fully operational.

Thanks for your patience, and we apologize for the disruption.
Posted Feb 06, 2023 - 12:56 GMT-03:00
This incident affected: WebStore.