Salesforce.com has revealed that a bug in the firmware of its storage arrays was behind last week’s data loss incident. The mess started in the company’s Washington data centre on May 9th, when admins noticed “a circuit breaker responsible for controlling power into the data center had failed.” “The team engaged the circuit vendor who began the process of replacing the failed breaker. Multiple redundant power systems had not engaged, which led to power failures at the computer system level.” That mess took the company’s NA14 instance offline, so it took steps to move it into a Chicago data centre. The move worked, but not long afterwards database performance dived.
View full story
ORIGINAL SOURCE: The Register