Degraded search experience
Incident Report for VTEX
Postmortem

“Between 2:02 PM to 6:50 PM UTC, shoppers experienced degraded search experience on Stores using Intelligent Search. You can read further details here: https://io.vtex.com.br/incident-report/2022-10-12-Intelligent_Search_outage.pdf

Posted Oct 21, 2022 - 14:06 GMT-03:00

Resolved
This incident has been resolved.
Posted Oct 12, 2022 - 19:13 GMT-03:00
Update
We are reviewing and doing a double check on the new backend search. The current error rate is still very low and at nominal values. Also, we are working to normalize the indexing latency, now that should be back in 40 minutes.
Posted Oct 12, 2022 - 18:42 GMT-03:00
Update
We noticed that product indexing is affected and we expect a latency in product indexing of roughly 2 hours (from the time the product is submitted for indexing to it being available on the store). We are working to reduc/normalizing this latency, which should be ok in 2h time as well
Posted Oct 12, 2022 - 17:25 GMT-03:00
Monitoring
While we are still doing data movement operations and using fallback search mechanisms, the current error rate is very low and at nominal values. User experience should be either ok (for stores that we already did data movement) or slightly degraded in search quality/hits (for stores that we changed the search mechanism). Overall latency and error rates should be ok for all. We are still working on our data movement operations which will take some considerably time (hours scale)
Posted Oct 12, 2022 - 16:23 GMT-03:00
Update
We are still working on mitigating this issue by using a different search backend.
Posted Oct 12, 2022 - 15:22 GMT-03:00
Update
We continue to work on sorting this issue. The impact was already minimised for a group of stores. As data movement is necessary this is taking a while. Meanwhile we are exploring further mitigating this issue by using a different search backend. This will provide a non ideal search experience while we work in re-establishing the main search mechanism
Posted Oct 12, 2022 - 14:17 GMT-03:00
Update
The problem has already been identified and the engineers are working on the solution
Posted Oct 12, 2022 - 12:00 GMT-03:00
Identified
Intelligent Search outage on uncached requests since 11:20 AM BRT
Posted Oct 12, 2022 - 11:55 GMT-03:00
This incident affected: WebStore.