Write-up published
Read our write-up of this incident at https://io.vtex.com.br/incident-report/Public-Incident-Report-Partial-outage-in-Checkout-314.pdf
Resolved
This incident has been resolved. We will provide further information in a public incident report.
Summary
On Nov 28, 2023, from 15:51 to 16:17 UTC, several stores experienced error ORD069 "Invalid contact" at Checkout.
Our global sales flow was partially affected for 26 minutes during this incident.
We apologize for any inconvenience this may have caused.
Timeline
At 14:38 UTC, our team started the gradual deployment of a new version of Checkout.
At 16:08 UTC, our incident response team was notified of an increase in ORD069 "Invalid contact" errors.
At 16:09 UTC, we started reverting Checkout to the previous version.
At 16:17 UTC, most service health indicators were back to nominal levels.
At 16:29 UTC, we completed reverting Checkout to the previous version and the incident was fully mitigated.
Monitoring
From 15:51 to 16:17 UTC some stores experienced error ORD069 "Invalid contact" at Checkout. The fix for the issue in Checkout has been implemented and stores should not longer be experiencing that symptom.
Our incident response team is monitoring to guarantee that normal platform behavior is fully reestablished.
We will send an additional update in the next 30 minutes, or as soon as we have more information to share.
Monitoring
We experienced issues with Checkout. We will bring more information soon.