Degraded search experience
Affected components

No components marked as affected

Updates

Write-up published

Read it here

Resolved

“Between 2:02 PM to 6:50 PM UTC, shoppers experienced degraded search experience on Stores using Intelligent Search. You can read further details here: https://io.vtex.com.br/incident-report/2022-10-12-Intelligent\_Search\_outage.pdf

Fri, Oct 21, 2022, 05:06 PM
1 week earlier...

Resolved

This incident has been resolved.

Wed, Oct 12, 2022, 10:13 PM
31m earlier...

Monitoring

We are reviewing and doing a double check on the new backend search. The current error rate is still very low and at nominal values. Also, we are working to normalize the indexing latency, now that should be back in 40 minutes.

Wed, Oct 12, 2022, 09:42 PM
1h earlier...

Monitoring

We noticed that product indexing is affected and we expect a latency in product indexing of roughly 2 hours (from the time the product is submitted for indexing to it being available on the store). We are working to reduc/normalizing this latency, which should be ok in 2h time as well

Wed, Oct 12, 2022, 08:25 PM
1h earlier...

Monitoring

While we are still doing data movement operations and using fallback search mechanisms, the current error rate is very low and at nominal values. User experience should be either ok (for stores that we already did data movement) or slightly degraded in search quality/hits (for stores that we changed the search mechanism). Overall latency and error rates should be ok for all. We are still working on our data movement operations which will take some considerably time (hours scale)

Wed, Oct 12, 2022, 07:23 PM
1h earlier...

Identified

We are still working on mitigating this issue by using a different search backend.

Wed, Oct 12, 2022, 06:22 PM
1h earlier...

Identified

We continue to work on sorting this issue. The impact was already minimised for a group of stores. As data movement is necessary this is taking a while. Meanwhile we are exploring further mitigating this issue by using a different search backend. This will provide a non ideal search experience while we work in re-establishing the main search mechanism

Wed, Oct 12, 2022, 05:17 PM
2h earlier...

Identified

The problem has already been identified and the engineers are working on the solution

Wed, Oct 12, 2022, 03:00 PM

Identified

Intelligent Search outage on uncached requests since 11:20 AM BRT

Wed, Oct 12, 2022, 02:55 PM