This incident has been resolved. We will provide further information in a public incident report.
Summary
On November 10, from 12:28 PM UTC to 1:06 PM UTC, shoppers experienced issues when completing a purchase.
Our global sales flow was affected for 38 minutes during this incident. One of our Payments services, Payment Options, became unhealthy during deployment, and we made rollback maneuvers to recover the platform fully.
We apologize for any inconvenience this may have caused.
Timeline
At 12:09 PM UTC, our team started deploying one of Payments Gateway’s component (Payment Options).
At 12:19 PM UTC, our team was alarmed by the system due to an increased error rate on this module and started the rollback process.
At 12:29 PM UTC, the rollback didn’t work as expected, and we declared it an incident to respond properly.
At 12:51 PM UTC, we started moving the traffic from Payment Options to the Payments Gateway.
At 1:03 PM UTC, the remaining traffic was completely moved to Payments Gateway.
At 1:06 PM UTC, we observed that shoppers were able to place orders again normally. The incident was fully mitigated.