Extended delays in product reindexing queue
Affected components

No components marked as affected

Updates

Write-up published

Read it here

Resolved

From July 31 21:00 UTC to August 1 16:59 UTC the platform presented extended delays in the product reindexing queue for Intelligent Search.

It did not have any impact in the sales flow, but caused delayed updates in search and product listing pages \(e.g. promotions might not display the most up-to-date information in these pages\).

We apologize for any inconvenience this may have caused.

Timeline

At 13:15 UTC, we started investigating extended delays when reindexing products in Intelligent Search.

At 13:24 UTC, we identified processing delays in our indexing queue started on July 31 at 21:00 UTC. This processing lag caused our indexing queue to grow until it started to recover automatically through our autoscaling systems on August 1 07:00 UTC.

At 14:30 UTC, we confirmed a consistent improvement in the observed symptoms, as product accumulated in our indexing queues were processed with increased processing provided by our autoscaling systems since August 1 07:00 UTC.

At 16:33 UTC, we continued monitoring our indexing queue, and observed that most products were indexing normally then.

At 16:59 UTC, we confirmed that our indexing queue was fully normalized.

Tue, Aug 1, 2023, 08:11 PM
3h earlier...

Resolved

The issue with Intelligent Search is solved. Our sincere apologies for the disruption in our services today. We appreciate you bearing with us in the meantime. We'll provide further information shortly.

Tue, Aug 1, 2023, 04:59 PM
26m earlier...

Monitoring

We are still monitoring our indexing queue. Most products should be indexing normally at this point.

We expect all indexing to be fully normalized within the next 30 minutes.

Tue, Aug 1, 2023, 04:33 PM
2h earlier...

Monitoring

We can confirm an improvement on the extended delays in our platform since 07:00 UTC.

We expect products accumulated in our queues to be processed and indexing to be fully normalized within the next 2 hours.

This does not have any impact in the sales flow, but might cause delayed updates in search and product listing pages (e.g. promotions might not display the most up-to-date information in these pages)

Tue, Aug 1, 2023, 02:30 PM
1h earlier...

Identified

We have identified processing delays in our indexing queue started on July 31 at 21:00 UTC.

This processing lag caused our indexing queue to grow until it started to recover automatically through our autoscaling systems on August 1 07:00 UTC.

Tue, Aug 1, 2023, 01:24 PM

Investigating

We are investigating extended delays when reindexing products in Intelligent Search.

Tue, Aug 1, 2023, 01:15 PM