This incident has been resolved. We will not provide further information in a public incident report.
Summary
On Jan 19, 2024, from 10:30 to 12:02 UTC, shoppers navigating in Store Framework storefronts and other applications depending on VTEX IO platform infrastructure experienced 5xx errors.
Our sales flow for Store Framework storefronts was affected for 49 minutes during this incident, across 4 different time windows.
We apologize for any inconvenience this may have caused.
Timeline
At 10:30 UTC, there was an increase in 5xx errors in the VTEX IO platform.
At 10:33 UTC, our incident response team was notified of the issue and started investigating.
At 10:45 UTC, our autoscaling and self-healing mechanisms mitigated the impact temporarily. The incident response team continued monitoring and investigating the root cause of the errors. At this point, we had already observed that a surge of requests to our platform could have been the trigger.
At 11:15 UTC, there was another surge of requests to our platform, triggering another increase in 5xx errors in the VTEX IO platform. Our autoscaling and self-healing mechanisms continued to respond as expected.
At 11:20 UTC, our incident response confirmed the hypothesis that we were experiencing a DDoS attack. The team started implementing mitigation actions against it.
At 11:36 UTC, our autoscaling and self-healing mechanisms mitigated the impact temporarily.
At 11:46 UTC, there was another surge of requests to our platform, triggering another increase in 5xx errors in the VTEX IO platform. Our incident response team continued to implement mitigation actions focused on the DDoS attack.
At 12:02 UTC, the incident was fully mitigated.