This incident has been resolved. We will not provide further information in a public incident report.
Summary
On Jan 19, 2024, from 10:30 to 12:02 UTC, shoppers navigating in Store Framework storefronts and other applications depending on VTEX IO platform infrastructure experienced 5xx errors.
Our sales flow for Store Framework storefronts was affected for 49 minutes during this incident, across 4 different time windows.
We apologize for any inconvenience this may have caused.
Timeline
At 10:30 UTC, there was an increase in 5xx errors in the VTEX IO platform.
At 10:33 UTC, our incident response team was notified of the issue and started investigating.
At 10:45 UTC, our autoscaling and self-healing mechanisms mitigated the impact temporarily. The incident response team continued monitoring and investigating the root cause of the errors. At this point, we had already observed that a surge of requests to our platform could have been the trigger.
At 11:15 UTC, there was another surge of requests to our platform, triggering another increase in 5xx errors in the VTEX IO platform. Our autoscaling and self-healing mechanisms continued to respond as expected.
At 11:20 UTC, our incident response confirmed the hypothesis that we were experiencing a DDoS attack. The team started implementing mitigation actions against it.
At 11:36 UTC, our autoscaling and self-healing mechanisms mitigated the impact temporarily.
At 11:46 UTC, there was another surge of requests to our platform, triggering another increase in 5xx errors in the VTEX IO platform. Our incident response team continued to implement mitigation actions focused on the DDoS attack.
At 12:02 UTC, the incident was fully mitigated.
Resolved
This incident has been resolved. We will not provide further information in a public incident report.
Summary
On Jan 19, 2024, from 10:30 to 12:02 UTC, shoppers navigating in Store Framework storefronts and other applications depending on VTEX IO platform infrastructure experienced 5xx errors.
Our sales flow for Store Framework storefronts was affected for 49 minutes during this incident, across 4 different time windows.
We apologize for any inconvenience this may have caused.
Timeline
At 10:30 UTC, there was an increase in 5xx errors in the VTEX IO platform.
At 10:33 UTC, our incident response team was notified of the issue and started investigating.
At 10:45 UTC, our autoscaling and self-healing mechanisms mitigated the impact temporarily. The incident response team continued monitoring and investigating the root cause of the errors. At this point, we had already observed that a surge of requests to our platform could have been the trigger.
At 11:15 UTC, there was another surge of requests to our platform, triggering another increase in 5xx errors in the VTEX IO platform. Our autoscaling and self-healing mechanisms continued to respond as expected.
At 11:20 UTC, our incident response confirmed the hypothesis that we were experiencing a DDoS attack. The team started implementing mitigation actions against it.
At 11:36 UTC, our autoscaling and self-healing mechanisms mitigated the impact temporarily.
At 11:46 UTC, there was another surge of requests to our platform, triggering another increase in 5xx errors in the VTEX IO platform. Our incident response team continued to implement mitigation actions focused on the DDoS attack.
At 12:02 UTC, the incident was fully mitigated.
Monitoring
The fix for the issue in VTEX IO has been implemented.
Shoppers should no longer be experiencing 5xx errors in storefronts..
Our incident response team is monitoring to guarantee that normal platform behavior is fully reestablished.
We will send an additional update in the next 30 minutes, or as soon as we have more information to share.
Investigating
We continue to investigate an issue in VTEX IO.
The evidence collected so far by our incident response team indicate that we are going through a DDoS attack. Mitigation actions are being applied to reduce impact on our platform.
We will send an additional update in the next 30 minutes, or as soon as we have more information to share.
Investigating
We are currently investigating an issue in VTEX IO.
Shoppers may be experiencing 5xx errors in storefronts.
Our incident response team is working to identify the root cause and implement a solution.
We will send an additional update in the next 30 minutes, or as soon as we have more information to share.