VTEX Status

Dec 3, 2020

No incidents reported today.

Dec 2, 2020

No incidents reported.

Dec 1, 2020

No incidents reported.

Nov 30, 2020

Resolved - Between 22:45 and 23:29 UTC, we experienced a degraded performance in our search and navigation, impacting 5% of our stores.
We are working to avoid this issue in the future. The service is now operating normally.

You can read the full incident report here: https://io.vtex.com.br/incident-report/2020-11-30-Incident_30_Nov_2020_Timeouts_on_Intelligent_Search.pdf
Nov 30, 21:56 GMT-03:00

Monitoring - We can confirm an improvement in the elevated error rates in our platform. We are monitoring the result of our actions.
Nov 30, 20:53 GMT-03:00

Update - We are continuing to work towards full resolution of this issue. We continue to work on recovery.
Nov 30, 20:40 GMT-03:00

Identified - We have identified an increasing elevated error rates on search and navigation of about 5% our customers. We are already working to resolve this issue.
Nov 30, 20:39 GMT-03:00

Nov 29, 2020

No incidents reported.

Nov 28, 2020

No incidents reported.

Nov 27, 2020

Delay to update orders status

Resolved - Between 22h15 UTC and 02h15 UTC, administrative users faced delays to see the correct status of orders, both in the administrative panel and through the Orders APIs.

We can now confirm that all systems are normalized and our order statuses are up-to-date. We will keep monitoring and soon will be publishing a complete report.

You can read the full incident report here: https://io.vtex.com.br/incident-report/2020-11-27-Incident_27_Nov_2020_Degraded_Payment_Gateway_Service.pdf
Nov 27, 23:27 GMT-03:00

Monitoring - We can confirm an improvement in the systems and soon all orders status will be up to date. We are monitoring and soon will get back with a complete report.
Nov 27, 23:06 GMT-03:00

Identified - The issue has been identified and a fix is being implemented.
Nov 27, 22:16 GMT-03:00

Investigating - Due to the reprocessing of purchases, we have identified a delay in updating the order status in the order workflow. As soon as the operation is normalized, we’ll post new updates.
Nov 27, 22:16 GMT-03:00

Degraded Performance

Resolved - Due to an infrastructure issue with our caching servers, our gateway has experienced higher latency than usual, generating increased time-out in orders. In order to reestablish normal operations, we restarted our cache servers, and the operation returned to normal.

We are working to avoid this issue in the future, and soon we will share a complete Incident Report with further actions. The service is now operating normally.

You can read the full incident report here: https://io.vtex.com.br/incident-report/2020-11-27-Incident_27_Nov_2020_Degraded_Payment_Gateway_Service.pdf
Nov 27, 20:29 GMT-03:00

Monitoring - We have identified degraded performance in our payment gateway services and our team has performed changes in our infrastructure. We can confirm an improvement in the error rates. We are now monitoring the result of our actions.
Nov 27, 20:10 GMT-03:00

Investigating - We’re currently investigating an increase in latency on one of our services. As soon as the operation is normalized, we’ll post new updates.
Nov 27, 19:25 GMT-03:00

Nov 26, 2020

Elevated Errors

Resolved - From 10h40 PM to 10h55 PM UTC, a regular customer trying to shop on VTEX stores would not be able to navigate or complete any orders. We are working to avoid this issue in the future. The service is now operating normally.

You can read the full incident report here: https://io.vtex.com.br/incident-report/2020-11-25-Incident_25_Nov_2020_Sales_down.pdf
Nov 26, 00:26 GMT-03:00

Monitoring - We can confirm an improvement in the elevated error rates in our platform. We are monitoring the result of our actions.
Nov 25, 20:20 GMT-03:00

Nov 25, 2020

Delays in orders listing and searching and catalog indexing.

Resolved - Between 01:26 PM and 10:10 PM UTC, we experienced a degraded performance in our infrastructure, impacting our admin console and indexing changes in the storefront. We are working to avoid this issue in the future, and soon we will share a complete Incident Report with further actions. The service is now operating normally.

You can read the full incident report here: https://io.vtex.com.br/incident-report/2020-11-25-Incident_25_Nov_2020_Catalog_and_orders_indexing_and_insights_%20degraded.pdf
Nov 25, 21:59 GMT-03:00

Monitoring - All of our services are operational again. We still have delays on the catalog and orders list services that are gradually normalizing. We are actively monitoring those services.
Nov 25, 19:10 GMT-03:00

Update - We reestablished the operation of our orders list services. Due to the downtime, in the upcoming hours our systems will be processing all orders. Because of that, the service will still face delays — until it returns to its normal behavior.
Nov 25, 18:31 GMT-03:00

Update - We are currently releasing new servers, unaffected by the original issue, to ensure the re-establishment of the order listing API and Admin, as well as our insights tool. As soon as the operation is normalized, we’ll post new updates.
Nov 25, 17:58 GMT-03:00

Update - We are currently working on re-establishing the order listing page in our admin console and API. As soon as the operation is normalized, we’ll post another update here.
Nov 25, 16:42 GMT-03:00

Update - We have reestablished the catalog indexing system. We are now processing all pending changes, which may cause additional delays until it's finished. In the upcoming hours, all product indexing will be working as usual. We are still working to reestablish the healthy of our order listing.
Nov 25, 16:03 GMT-03:00

Update - We have identified that the same infrastructure issue impacts the indexing of new products and generates inconsistencies in the search filtering by price. There are no impacts on the price shown in the product details pages and other search results.
Nov 25, 15:39 GMT-03:00

Identified - We have identified infrastructure issues affecting the platform.

The issue is causing delays in our order listing and searching through our API and admin console. The integration APIs (feed and hook) are working normally. This issue does not impact sessions, sales, or order processing.
We are also experiencing issues with our Insights admin tool due to the same infrastructure problems.

We are currently working to resolve the issue.
Nov 25, 11:53 GMT-03:00

Nov 24, 2020

No incidents reported.

Nov 23, 2020

No incidents reported.

Nov 22, 2020

No incidents reported.

Nov 21, 2020

No incidents reported.

Nov 20, 2020

Elevated Errors

Resolved - Between 11:30 AM and 11:39 AM UTC, we experienced degraded sales in our platform. We are working to avoid this issue in the future. The service is now operating normally.

You can read the full incident report here: https://io.vtex.com.br/incident-report/2020-11-20-Incident_20_Nov_2020_Degraded_checkout.pdf
Nov 20, 08:30 GMT-03:00

Elevated Errors

Resolved - Between 09:48 PM and 10:34 PM UTC-3 (BRT), we experienced elevated error rates in our platform. We are working to avoid this issue in the future. This incident has been resolved and services are now operating normally.

You can read our incident report here:
https://io.vtex.com.br/incident-report/2020-11-20-Incident_20_Nov_2020_Sales_down.pdf
Nov 20, 00:46 GMT-03:00

Monitoring - We can confirm an improvement in the elevated error rates in our platform. We are monitoring the result of our actions.
Nov 19, 22:48 GMT-03:00

Identified - We are now recovering from the increased error rates. We continue to work towards full recovery.
Nov 19, 22:39 GMT-03:00

Update - We are continuing to investigate increased error rates in some parts of our platform.
Nov 19, 22:21 GMT-03:00

Investigating - We are investigating increased error rates in our platform.
Nov 19, 22:04 GMT-03:00

Nov 19, 2020