This incident has been resolved. We will provide further information in a public incident report.
Summary
On November 10, from 12:28 PM UTC to 1:06 PM UTC, shoppers experienced issues when completing a purchase.
Our global sales flow was affected for 38 minutes during this incident. One of our Payments services, Payment Options, became unhealthy during deployment, and we made rollback maneuvers to recover the platform fully.
We apologize for any inconvenience this may have caused.
Timeline
At 12:09 PM UTC, our team started deploying one of Payments Gateway’s component (Payment Options).
At 12:19 PM UTC, our team was alarmed by the system due to an increased error rate on this module and started the rollback process.
At 12:29 PM UTC, the rollback didn’t work as expected, and we declared it an incident to respond properly.
At 12:51 PM UTC, we started moving the traffic from Payment Options to the Payments Gateway.
At 1:03 PM UTC, the remaining traffic was completely moved to Payments Gateway.
At 1:06 PM UTC, we observed that shoppers were able to place orders again normally. The incident was fully mitigated.
No components marked as affected
Resolved
This incident has been resolved. We will provide further information in a public incident report.
Summary
On November 10, from 12:28 PM UTC to 1:06 PM UTC, shoppers experienced issues when completing a purchase.
Our global sales flow was affected for 38 minutes during this incident. One of our Payments services, Payment Options, became unhealthy during deployment, and we made rollback maneuvers to recover the platform fully.
We apologize for any inconvenience this may have caused.
Timeline
At 12:09 PM UTC, our team started deploying one of Payments Gateway’s component (Payment Options).
At 12:19 PM UTC, our team was alarmed by the system due to an increased error rate on this module and started the rollback process.
At 12:29 PM UTC, the rollback didn’t work as expected, and we declared it an incident to respond properly.
At 12:51 PM UTC, we started moving the traffic from Payment Options to the Payments Gateway.
At 1:03 PM UTC, the remaining traffic was completely moved to Payments Gateway.
At 1:06 PM UTC, we observed that shoppers were able to place orders again normally. The incident was fully mitigated.
Monitoring
We continue to ensure the stability of our operations and our team is working diligently to prevent this issue from happening again.
We will send an additional update in the next 30 minutes, or as soon as we have more information to share.
Monitoring
We are operating normally and aren't observing any impact on sales flow. We are being extra cautious, though, deciding to monitor longer the issue to prevent this issue from happening again.
We will send an additional update in the next 30 minutes, or as soon as we have more information to share.
Monitoring
The fix for the issue in Payments Gateway has been applied, and we have recovered the sales flow. Shoppers should be able to place orders normally.
Our incident response team is monitoring to guarantee that normal platform behavior is fully reestablished.
We will send an additional update in the next 30 minutes, or as soon as we have more information to share.
Identified
We have identified an issue on our Payments Gateway.
Shoppers are experiencing issues when completing a purchase.
Our incident response team is working to fix the issue.
We will send an additional update in the next 30 minutes, or as soon as we have more information to share.
