This incident has been resolved.

On the evening of October 18, 2023, our alert system detected an issue impacting the order workflow update process. While this issue didn't directly impact sales, we acted promptly to mitigate the impact on the order fulfillment process. Here's a detailed timeline of the incident:

• 21:47 UTC: An issue within one of our system components responsible for updating the order workflow was detected. Our monitoring systems triggered alarms, prompting the automatic recovery system to kick in.

• 22:01 UTC: The automatic recovery system partially restored the affected system, and we began to see improvements in the order workflow update process.

• 22:12 UTC: However, we identified a problem with one of the balancers, necessitating a manual recovery process to ensure a comprehensive fix.

• 22:28 UTC: Manual recovery actions were initiated, and their impact became noticeable.

• 22:37 UTC: Substantial progress was made, with a noticeable recovery in most order workflow updates.

• 22:59 UTC: The order workflow update process was fully recovered.

During this incident, changes in order workflow were temporarily backlogged. To address this backlog, we took action to index these changes, ensuring no data was lost.

• 23:35 UTC: All order workflow changes had been successfully indexed, and we continued to closely monitor the systems for approximately 40 minutes to ensure the incident was entirely resolved.

The incident has been addressed, and the order workflow update process is back to normal. Our team is working diligently to prevent such occurrences in the future.

All orders have been indexed, and merchants should no longer experience delays on order flow at the moment.

We have an estimated time of 30 minutes to finish the indexing.

All order flow operations have been fully restored. Currently, there is a delay in indexing. At this moment, the delay is around 55 minutes, with the maximum delay reaching one hour and five minutes.

We applied a fix, and the order flow is returning to normal. We will provide more information shortly.

We are continuing to work on a fix for this issue.