This incident has been resolved. We will provide further information in a public incident report.
Summary
On Mar 14, 2024, from 14:46 to 15:33 UTC, systems and integrations that have Master Data as a dependency (such as some Logistics/B2B apps and services) experienced a higher than usual 5xx error rate.
Our global sales flow was partially affected for 20 minutes during this incident. After that, degraded performance was observed for 27 minutes.
We apologize for any inconvenience this may have caused.
Timeline
At 14:46 UTC, the team responsible for Master Data identified that a recent configuration change was unsucessful. They immediately started mitigation actions to revert the configuration back to its original state.
At 14:48 UTC, the configuration was reverted back to its original state. However, some instances lingered in an unhealthy state even after the configuration was reverted.
At 14:53 UTC, our incident response team was notified of the issue.
At 15:05 UTC, the team identified that the instances lingered in an unhealthy state due to corrupted cache. The team started discussing potential mitigation actions.
At 15:19 UTC, the team started identifying and terminating unhealthy instances as a mitigation action.
At 15:33 UTC, the team completed all mitigation actions and normal platform behavior was reestablished.