This incident has been resolved. We will provide further information in a public incident report.
Summary
On November 8, 2023, from 10:28 PM to 11:12 PM UTC, Storefronts and Admin experienced a slight increase in latency and intermittent timeout errors. Approximately only 15% of our global sessions were affected by this issue, and as a side effect, 6% of the orders were affected.
This degraded performance affected the global sales and admin flows for 44 minutes during this incident.
We apologize for any inconvenience this may have caused.
Timeline
At 10:31 PM UTC, our incident response team was notified of the issue.
At 10:43 PM UTC, we identified the issue and started the mitigation actions.
At 10:53 PM UTC, we applied actions that improved our overall metrics.
At 11:01 PM UTC, we applied additional actions to fully mitigate the issue.
At 11:12 PM UTC, we started monitoring our platform after fully recovering our global sessions and admin.
At 11:26 PM UTC, we declared that the incident was fully mitigated.
No components marked as affected
Resolved
This incident has been resolved. We will provide further information in a public incident report.
Summary
On November 8, 2023, from 10:28 PM to 11:12 PM UTC, Storefronts and Admin experienced a slight increase in latency and intermittent timeout errors. Approximately only 15% of our global sessions were affected by this issue, and as a side effect, 6% of the orders were affected.
This degraded performance affected the global sales and admin flows for 44 minutes during this incident.
We apologize for any inconvenience this may have caused.
Timeline
At 10:31 PM UTC, our incident response team was notified of the issue.
At 10:43 PM UTC, we identified the issue and started the mitigation actions.
At 10:53 PM UTC, we applied actions that improved our overall metrics.
At 11:01 PM UTC, we applied additional actions to fully mitigate the issue.
At 11:12 PM UTC, we started monitoring our platform after fully recovering our global sessions and admin.
At 11:26 PM UTC, we declared that the incident was fully mitigated.
Monitoring
We continue to ensure the stability of our operations and our team is working diligently to implement preventive measures.
We will send an additional update in the next 30 minutes, or as soon as we have more information to share.
Monitoring
Although we are operating normally and we aren't observing any impact in sessions and Admin, we are being extra cautious. So, we've decided to monitor longer and we are applying additional actions to prevent this issue from happening again.
We will send an additional update in the next 30 minutes, or as soon as we have more information to share.
Monitoring
The fix for the issue in IO Platform has been effectively implemented.
Shoppers and merchants should not longer be experiencing high latency and errors in their navigation flow .
Our incident response team is monitoring to guarantee that normal platform behavior is fully reestablished.
We will send an additional update in the next 30 minutes, or as soon as we have more information to share.
Identified
We have identified the trigger of the issue in IO platform.
Our incident response team estimates that the fix will be implemented in the next 15 minutes.
We will send an additional update in the next 30 minutes, or as soon as we have more information to share.
