Between 01:26 PM and 10:10 PM UTC, we experienced a degraded performance in our infrastructure, impacting our admin console and indexing changes in the storefront. We are working to avoid this issue in the future, and soon we will share a complete Incident Report with further actions. The service is now operating normally.

You can read the full incident report here: https://io.vtex.com.br/incident-report/2020-11-25-Incident_25_Nov_2020_Catalog_and_orders_indexing_and_insights_%20degraded.pdf

All of our services are operational again. We still have delays on the catalog and orders list services that are gradually normalizing. We are actively monitoring those services.

We reestablished the operation of our orders list services. Due to the downtime, in the upcoming hours our systems will be processing all orders. Because of that, the service will still face delays — until it returns to its normal behavior.

We are currently releasing new servers, unaffected by the original issue, to ensure the re-establishment of the order listing API and Admin, as well as our insights tool. As soon as the operation is normalized, we’ll post new updates.

We are currently working on re-establishing the order listing page in our admin console and API. As soon as the operation is normalized, we’ll post another update here.

We have reestablished the catalog indexing system. We are now processing all pending changes, which may cause additional delays until it's finished. In the upcoming hours, all product indexing will be working as usual. We are still working to reestablish the healthy of our order listing.

We have identified that the same infrastructure issue impacts the indexing of new products and generates inconsistencies in the search filtering by price. There are no impacts on the price shown in the product details pages and other search results.