Post-mortem about VTEX IO and administrative environment
Resolved

Between 06/09/2018 and 13/09/2018 we experienced intermitent errors in the VTEX IO service, currently responsible for serving your admins and a few stores.

The errors caused a few events of downtime in the admins about once or twice a day, normally around 9:00 AM UTC-3 (Brasília) when the traffic increased in the platform.

The investigation lasted the whole week and it was found that the problem was caused by a combination of incidents in IO infrastructure services and an app that was running in IO without proper monitoring.

The issues in our infrastructure have already been fixed and we have worked with the app developers to fix its problems as well.We are currently working to avoid any other issues with the related services and to minimize the impact that an app could cause in the IO platform.

Thu, Sep 20, 2018, 07:30 PM
5 years ago
Affected components

No components marked as affected

Updates

Resolved

Between 06/09/2018 and 13/09/2018 we experienced intermitent errors in the VTEX IO service, currently responsible for serving your admins and a few stores.

The errors caused a few events of downtime in the admins about once or twice a day, normally around 9:00 AM UTC-3 (Brasília) when the traffic increased in the platform.

The investigation lasted the whole week and it was found that the problem was caused by a combination of incidents in IO infrastructure services and an app that was running in IO without proper monitoring.

The issues in our infrastructure have already been fixed and we have worked with the app developers to fix its problems as well.We are currently working to avoid any other issues with the related services and to minimize the impact that an app could cause in the IO platform.

Thu, Sep 20, 2018, 07:30 PM