Intermittent errors in Catalog and Search APIs
11:35 PM
This incident has been resolved. We will provide further information in a public incident report.
Summary
From Dec 16, 2024 at 17:30 UTC to Dec 17, 2024 at 23:00 UTC, some accounts experienced errors in requests to Catalog and Search APIs. This affected shopper navigation and order placement under very specific circumstances.
The issue was triggered by a recent deployment of a new service focused on improving the efficiency of Catalog data retrieval which, as an unexpected side-effect, temporarily caused data corruption of some SKUs that were recently updated.
Our global sales flow was unaffected during this incident. Only shopper navigation and order placement flows involving affected SKUs were impacted as a result, which occurred very rarely.
We apologize for any inconvenience this may have caused.
Timeline
[Dec 16, 2024]
At 17:30 UTC, our team deployed a new service focused on improving the efficiency of Catalog data retrieval.
[Dec 17, 2024]
At 13:56 UTC, our incident response team was notified of increased error rates returned to other internal services.
At 14:46 UTC, we identified that the issue was connected to the new service and deactivated it.
At 15:00 UTC, we observed significant improvements in the reported error rates.
At 16:30 UTC, we received reports from merchants that seemed to be connected to lingering side-effects of the issue. We started investigating the issue in the reported cases.
At 18:15 UTC, we identified that under very specific circumstances, certain SKUs may have had their data temporarily corrupted in our Catalog/Search indices. Our incident response started applying mitigation actions.
At 23:00 UTC, the incident was fully mitigated.