This incident has been resolved. We will provide further information in a public incident report.
Summary
On Dec, 15, 2023, from 01:58 UTC to 02:40 UTC, all accounts experienced partial outage on sales flow.
Our global sales flow was affected for 42 minutes during this incident.
We apologize for any inconvenience this may have caused.
Timeline
At 01:58 UTC, our system detected an issue with the RNB system, and the team was alerted.
The team responded promptly and identified at 02:03 UTC that some machines in the system were unhealthy, leading to an increase in error 500.
The team began to take action by immediately deploying more machines as a mitigation strategy. Although the action was taken, it was not successful.
At this point, we initiated a disaster recovery maneuver, successfully partially restoring the sales flow at 02:37 UTC.
At 02:40 UTC, the incident was fully mitigated.