Extended delays in product reindexing queue
Incident Report for VTEX
Postmortem

From July 31 21:00 UTC to August 1 16:59 UTC the platform presented extended delays in the product reindexing queue for Intelligent Search.

It did not have any impact in the sales flow, but caused delayed updates in search and product listing pages (e.g. promotions might not display the most up-to-date information in these pages).

We apologize for any inconvenience this may have caused.

Timeline

At 13:15 UTC, we started investigating extended delays when reindexing products in Intelligent Search.

At 13:24 UTC, we identified processing delays in our indexing queue started on July 31 at 21:00 UTC. This processing lag caused our indexing queue to grow until it started to recover automatically through our autoscaling systems on August 1 07:00 UTC.

At 14:30 UTC, we confirmed a consistent improvement in the observed symptoms, as product accumulated in our indexing queues were processed with increased processing provided by our autoscaling systems since August 1 07:00 UTC.

At 16:33 UTC, we continued monitoring our indexing queue, and observed that most products were indexing normally then.

At 16:59 UTC, we confirmed that our indexing queue was fully normalized.

Posted Aug 03, 2023 - 01:11 UTC

Resolved
The issue with Intelligent Search is solved. Our sincere apologies for the disruption in our services today. We appreciate you bearing with us in the meantime. We'll provide further information shortly.
Posted Aug 01, 2023 - 16:59 UTC
Update
We are still monitoring our indexing queue. Most products should be indexing normally at this point.

We expect all indexing to be fully normalized within the next 30 minutes.
Posted Aug 01, 2023 - 16:33 UTC
Monitoring
We can confirm an improvement on the extended delays in our platform since 07:00 UTC.

We expect products accumulated in our queues to be processed and indexing to be fully normalized within the next 2 hours.

This does not have any impact in the sales flow, but might cause delayed updates in search and product listing pages (e.g. promotions might not display the most up-to-date information in these pages)
Posted Aug 01, 2023 - 14:30 UTC
Identified
We have identified processing delays in our indexing queue started on July 31 at 21:00 UTC.

This processing lag caused our indexing queue to grow until it started to recover automatically through our autoscaling systems on August 1 07:00 UTC.
Posted Aug 01, 2023 - 13:24 UTC
Investigating
We are investigating extended delays when reindexing products in Intelligent Search.
Posted Aug 01, 2023 - 13:15 UTC
This incident affected: Internal Modules.