High error rate on VTEX IO

Incident Report for VTEX

Resolved

This incident has been resolved. You can read the incident report here: https://io.vtex.com.br/incident-report/2022-02-03-High_error_rates_on_VTEX_IO.pdf

Posted Feb 04, 2022 - 01:07 GMT-03:00

Monitoring

We can confirm the errors stopped at 21:59 UTC ir our platform. We are monitoring the result of our actions and operating when necessary.

Posted Feb 03, 2022 - 22:40 GMT-03:00

Update

An operation to fully restore VTEX IO to get out of the mitigation state was successfully applied to a minority part of VTEX IO.

Posted Feb 03, 2022 - 20:47 GMT-03:00

Update

We've applied some operations in our infrastructure in order to mitigate the issue and errors are no more a concern.

Posted Feb 03, 2022 - 19:45 GMT-03:00

Identified

We have identified the root cause of the increased error rates. We are working to resolve this issue.

Posted Feb 03, 2022 - 19:35 GMT-03:00

Investigating

We are investigating increased error rates in VTEX IO.

Posted Feb 03, 2022 - 18:43 GMT-03:00