On Saturday July 16, Percona suffered a catastrophic failure of three disks on our primary web server. This was compounded by unexpected problems in recovery from our backups arising from staff changes. The net result is that several Percona web properties were offline from several hours to several days and that our cleanup continues. Interruptions in our provision of software downloads, documentation, and credit card transactions directly affected Percona customers and users of our software. However no customer data was compromised and customer access to our engineers via our customer service portal and online chat was uninterrupted. The web server that failed is completely separate from those that contain customer data.
The recovery lessons learned for us have been considerable and will be incorporated into our internal processes. Availability and performance of all of our websites is a top priority. On behalf of all of us at Percona, I apologize for the inconvenience this has caused for our users.
- Tom Basil, COO