Extented Delays in Orders Indexing
Resolved
Lasted for 5h

Between 11:27 AM and 05:08 PM UTC-3, we experienced delays in orders indexing. The orders search service was partly down after a failure in the cache layer and to avoid more damage some features was stopped. The index was slowly self-recovery and until it the system was throttled. After the recovery we start to index orders queue, which took more around 3 hours to process whole queue. The service is now operating normally.

Wed, Apr 22, 2020, 08:31 PM
4 years ago
Affected components

No components marked as affected

Updates

Resolved

Between 11:27 AM and 05:08 PM UTC-3, we experienced delays in orders indexing. The orders search service was partly down after a failure in the cache layer and to avoid more damage some features was stopped. The index was slowly self-recovery and until it the system was throttled. After the recovery we start to index orders queue, which took more around 3 hours to process whole queue. The service is now operating normally.

Wed, Apr 22, 2020, 08:31 PM
3h earlier...

Monitoring

We can confirm an improvement on the extended delays in our platform. We are monitoring the result of our actions.

Wed, Apr 22, 2020, 05:09 PM
26m earlier...

Identified

We recovered from the increased error rates in OMS administrative environment. But still working with extended delays in Orders indexing. We continue to work towards full recovery.

Wed, Apr 22, 2020, 04:42 PM
50m earlier...

Identified

We are experiencing an increased error rates in Administrative Environment of Orders Management System. We are working to resolve this issue.

Wed, Apr 22, 2020, 03:51 PM
13m earlier...

Identified

We have identified the root cause of the extended delays to indexing orders. We are working to resolve this issue.

Wed, Apr 22, 2020, 03:38 PM
10m earlier...

Investigating

We are experiencing delays in orders indexing. We are working to normalize service.

Wed, Apr 22, 2020, 03:28 PM