After extensive monitoring and troubleshooting of this incident for the last few days, we are confident that we have a good understanding of the incident contributing factors and are also confident with how effective are the mitigations put in place
We are considering this incident resolved at this point. We will continue to work to make improvements on the underlying infrastructure which contributed to it and/or was affected by it.
No components marked as affected
Resolved
After extensive monitoring and troubleshooting of this incident for the last few days, we are confident that we have a good understanding of the incident contributing factors and are also confident with how effective are the mitigations put in place
We are considering this incident resolved at this point. We will continue to work to make improvements on the underlying infrastructure which contributed to it and/or was affected by it.
Monitoring
All of our platform continues to operate normally since the last update. Both shopper and merchant experience should not be impacted.
We are monitoring our systems more closely since the last update, and will continue to do so throughout the weekend
Identified
VTEX Sales flow is back to normal.
We have been experiencing some instability in our platform with the pricing module throughout the day. Our team is dedicated to resolving this issue and monitoring the environment. At this moment, the sales flow is back to normal.
Period of Instability:Jun 16, 2023 - 07:09 to 07:34 BRTJun 16, 2023 - 09:43 to 10:15 BRTJun 16, 2023 - 15:03 to 15:18 BRT
Identified
There are ongoing issues currently impacting the shopper flow. Investigating
Monitoring
Product Indexing latency is normalized.
Identified
Product Indexing latency is converging. We expect to see it normalised around 40~50 minutes.
This means that changes to products by merchants should be visible to shoppers in 50 minutes or less
Identified
Sales are at normal levels, but product indexing is lagging. Expect to see latency between issuing updates in the catalog and those changes being live to shoppers. We are working on reducing such latency back to nominal levels
Identified
Sales and user navigation recovered in the last 15 minutes. We continue to work and investigate this issue
Identified
The issue is recurring, our previous mitigations were not fully effective. We are actively working on the issue
Monitoring
We are investigating an unexpected increase in errors in the pricing module between 10:09 UTC and 10:32 UTC. During that time window it is possible that shoppers were not able to complete their purchases.
We can confirm that the systems are back to normal as of 10:33 UTC, but we continue to investigate the contributing causes for this incident and monitor any weird behavior.
