Partial Degradation in Orders

Resolved

At 08:41 UTC, we observed instabilities in the Rates and Benefits module, resulting in partial order degradation. This led to errors and increased latency during the checkout process for users.

The response team identified the issue by 08:54 UTC and implemented mitigation actions for the Rates and Benefits module.

By 09:11 UTC, the effectiveness of these actions was confirmed, and order processing had fully recovered by 09:16 UTC. Subsequently, we kept monitoring our systems to ensure stability.

At 09:31 UTC, our comprehensive recovery efforts were verified without impacting shopper checkout and navigation flows or merchant operations (admin and APIs).

We remain committed to averting the recurrence of similar incidents in the future

Wed, Aug 9, 2023, 10:49 AM

(1 year ago)

Affected components

No components marked as affected

Updates

Resolved

At 08:41 UTC, we observed instabilities in the Rates and Benefits module, resulting in partial order degradation. This led to errors and increased latency during the checkout process for users.

The response team identified the issue by 08:54 UTC and implemented mitigation actions for the Rates and Benefits module.

By 09:11 UTC, the effectiveness of these actions was confirmed, and order processing had fully recovered by 09:16 UTC. Subsequently, we kept monitoring our systems to ensure stability.

At 09:31 UTC, our comprehensive recovery efforts were verified without impacting shopper checkout and navigation flows or merchant operations (admin and APIs).

We remain committed to averting the recurrence of similar incidents in the future

Wed, Aug 9, 2023, 10:49 AM

Monitoring

We are continuing to monitor for any further issues.

Wed, Aug 9, 2023, 09:49 AM(59 minutes earlier)

Monitoring

Our monitoring systems detected an abnormal level of 429 errors from the Rates and Benefits module. This issue generated a partial degradation in orders up to 75%. The response team immediately started the remediation, scaling the service with resource saturation.

Our actions were successful, and our systems are operating at normal levels. Shoppers are not being impacted at this moment. Orders are back at normal levels. We are monitoring the platform and will keep you updated.

Wed, Aug 9, 2023, 09:31 AM(17 minutes earlier)