VTEX Status

Oct 6, 2022

No incidents reported today.

Oct 5, 2022

No incidents reported.

Oct 4, 2022

No incidents reported.

Oct 3, 2022

No incidents reported.

Oct 2, 2022

No incidents reported.

Oct 1, 2022

No incidents reported.

Sep 30, 2022

No incidents reported.

Sep 29, 2022

No incidents reported.

Sep 28, 2022

Higher latency leading to timeout errors on admin environment

Resolved - This incident has been resolved.
Sep 28, 15:45 GMT-03:00

Monitoring - We are monitoring the result of our actions to ensure the admin environment is back to normal.
Sep 28, 15:36 GMT-03:00

Update - We have confirmed a return to normal behavior in our admin environment, with the exception of some rare flows. We are identifying those flows in order to restore their behavior.
Sep 28, 15:17 GMT-03:00

Identified - We identified the issue and noticed 2 things: we are receiving unusual traffic from particular accounts, as well our systems were not scaled enough to cope with such traffic. We scaled up to be able to absorb such traffic and are further investigating the surge. The admin should be working correctly now with potential increase in latency. Will keep updating our status
Sep 28, 14:40 GMT-03:00

Investigating - We have indeed noticed the higher latency regarding the admin environment. We are currently investigating the issue
Sep 28, 14:04 GMT-03:00

Sep 27, 2022

Partial global sales outage due to network failure across multiple failure domains

Postmortem - Read details
Sep 28, 17:38 GMT-03:00

Resolved - All systems nominal, we are fully recovered
Sep 27, 19:02 GMT-03:00

Update - We are still working on fixing the admin's increased latency and error rates and restoring it to normal behaviour. All other systems are operating normally.
Sep 27, 18:26 GMT-03:00

Update - We are still monitoring our admin environment's increased latency and restoring it to normal behavior. All other systems are back to normal.
Sep 27, 17:23 GMT-03:00

Monitoring - Our mitigation actions have restored the performance of the stores. We are monitoring the result of our actions and confirming that our administrative environment has returned to normal.
Sep 27, 16:56 GMT-03:00

Update - Our mitigation actions have restored great part of the stores behavior, and we expect to be back to normal soon.
Sep 27, 16:32 GMT-03:00

Update - We are continuing to work on restoring the behavior of the stores. We have seen some improvement in the webstore flow, but we aren't still up to normal levels.
Sep 27, 16:07 GMT-03:00

Update - We are continuing to work on a fix for this issue.
Sep 27, 15:24 GMT-03:00

Update - We can confirm that this partial outage is impacting sales, especially on VTEX IO stores, mostly due to customers being unable to navigate the store smoothly. We are working to restore our latency and normal behavior.
Sep 27, 14:46 GMT-03:00

Update - We have confirmed that a part of some stores' sales have been affected. We are continuing to apply mitigation measures.
Sep 27, 14:31 GMT-03:00

Identified - We've identified degraded performance in VTEX IO clusters. The admin panel is the most affected. Symptoms include high latency when serving pages and increased number of timeouts.

We are applying mitigation measures in order to reestablish normal behavior.
Sep 27, 14:04 GMT-03:00

Sep 26, 2022

No incidents reported.

Sep 25, 2022

No incidents reported.

Sep 24, 2022

Elevated errors on pricing module

Resolved - During 14:21 and 14:46 UTC, upwards of 50% of sales were impacted by a partial outage in our pricing module. We are preparing follow up actions in order for this incident to not happen again.
Sep 24, 13:47 GMT-03:00

Update - We are continuing to monitor for any further issues.
Sep 24, 11:59 GMT-03:00

Monitoring - We have identified elevated errors in our pricing module. Upwards of 50% of sales were impacted during 14:21 and 14:46 UTC.
We have applied mitigation measures and the module is now recovered. We are monitoring to confirm the incident was mitigated.
Sep 24, 11:53 GMT-03:00

Sep 23, 2022

No incidents reported.

Sep 22, 2022

No incidents reported.