Minor delay in payment processing of orders
Resolved
Lasted for 4h

This incident has been resolved. We will not provide further information in a public incident report.

Summary
On November 11, 2023, from 11:57 AM UTC to 12:20 PM UTC, due to an issue on Payments Gateway, merchants experienced delays in their order workflow. Our global sales flow was partially affected during 6 minutes of this incident. We apologize for any inconvenience this may have caused.

Timeline
At 11:58 UTC, our incident response team was notified of the issue. At 12:01 UTC, we started performing mitigation actions to reduce delays in the payment processing of orders.
At 14:30 UTC, the mitigation actions were successful, and we started the process of retrying pending transactions.
At 15:53 UTC, we managed to recover a considerable amount of transactions affected.
At 16:57 UTC, the process of recovering delayed transactions was concluded, and we started to apply the fix.
At 18:28 UTC, the incident was fully resolved.

Thu, Nov 9, 2023, 06:32 PM
5 months ago
Affected components

No components marked as affected

Updates

Resolved

This incident has been resolved. We will not provide further information in a public incident report.

Summary
On November 11, 2023, from 11:57 AM UTC to 12:20 PM UTC, due to an issue on Payments Gateway, merchants experienced delays in their order workflow. Our global sales flow was partially affected during 6 minutes of this incident. We apologize for any inconvenience this may have caused.

Timeline
At 11:58 UTC, our incident response team was notified of the issue. At 12:01 UTC, we started performing mitigation actions to reduce delays in the payment processing of orders.
At 14:30 UTC, the mitigation actions were successful, and we started the process of retrying pending transactions.
At 15:53 UTC, we managed to recover a considerable amount of transactions affected.
At 16:57 UTC, the process of recovering delayed transactions was concluded, and we started to apply the fix.
At 18:28 UTC, the incident was fully resolved.

Thu, Nov 9, 2023, 06:32 PM
45m earlier...

Monitoring

The incident is being monitored.

The fix for the issue in Payments Gateway has been implemented, and the accounts should no longer be experiencing delays in payment processing.

We will closely monitor the re-establishment of the system for the next hour.

Thu, Nov 9, 2023, 05:46 PM
49m earlier...

Identified

We finished working on the process to recover the delayed transactions and are now applying the fix. Although we have a small number of transactions delayed, we are committed to fix the issue.

We estimate the fix to be completed and fully operational in less than one hour, and we will update as soon as we have more information to share.

Thu, Nov 9, 2023, 04:57 PM
1h earlier...

Identified

We managed to recover a considerable amount of transactions affected during the incident. Right now, we are working on recovering the remaining delayed transactions.

We still estimate that payment processing delays will normalize in one hour or less.

We will send an additional update in the next hour or as soon as we have more information to share.

Thu, Nov 9, 2023, 03:53 PM
1h earlier...

Identified

The mitigation actions that we performed were successful. We've started the process of retrying pending transactions reducing delays in the payment processing of orders.

We still estimate that payment processing delays will normalize in one hour or less.

We will send an additional update in the next hour, or as soon as we have more information to share.

Thu, Nov 9, 2023, 02:48 PM
18m earlier...

Identified

We are currently performing mitigation actions that will reduce delays in the payment processing of orders.

The fix consists in retrying pending transactions and increasing our queue processing capacity. Our incident response team estimates that payment processing delays will normalize in one hour or less.

We will send an additional update in the next hour, or as soon as we have more information to share.

Thu, Nov 9, 2023, 02:30 PM
29m earlier...

Identified

Our team identified delays in payment processing on orders created between 11:57 AM and 12:20 PM UTC due to an issue on Payments Gateway, merchants can experience delays in their order workflow.

Our sales flow is operating normally, and orders outside this period are being processed normally.

Our incident response are working to fix the issue and normalize the payment workflow.

We will send an additional update in the next 30 minutes, or as soon as we have more information to share.

Thu, Nov 9, 2023, 02:00 PM