At 6:01 pm (UTC-3) we lost one node of search engine used by our Logistics module. Immediately we started the recovery process, which was completed at 11:20 pm (UTC-3), with full recovery of our service. During the recovery process, we noticed that some stores were partially affected. We already indentified the root cause of the issue and are working on definitive plan for that doesn't happen again in the future. In the meantime, as part of workround, we deployed two infrastructures of this search engine to be used as fallback if it's necessary. We'll update you during next week with the POSTMORTEM about the action plan to avoid this issue to occurs again
No components marked as affected
Resolved
At 6:01 pm (UTC-3) we lost one node of search engine used by our Logistics module. Immediately we started the recovery process, which was completed at 11:20 pm (UTC-3), with full recovery of our service. During the recovery process, we noticed that some stores were partially affected. We already indentified the root cause of the issue and are working on definitive plan for that doesn't happen again in the future. In the meantime, as part of workround, we deployed two infrastructures of this search engine to be used as fallback if it's necessary. We'll update you during next week with the POSTMORTEM about the action plan to avoid this issue to occurs again
Monitoring
We can confirm an improvement in the elevated error rates in our platform. We are monitoring the result of our actions.
Monitoring
At 06:01 PM (UTC-3) we lost one of our Search engine nodes used by our Logistics service and immediately started the recovery process which still in progress. It's next to finished but incident still impacting partially a small number of our customers. We continue to work towards full recovery of these customers
Identified
We are continuing to work towards full resolution of this issue. We continue to work on recovery.
Identified
We are continuing to work towards full resolution of this issue. We continue to work on recovery.
Identified
We are continuing to work towards full resolution of this issue. We continue to work on recovery.
Identified
We are continuing to work towards full resolution of this issue. We continue to work on recovery.
Investigating
We are investigating increased error rates in our platform.