There are times where you need to know exactly how much memory the mysqld server (or any other program) is using, where (i.e. for what function) it was allocated, how it got there (a backtrace, please!), and at what point in time the allocation happened. For example; you may have noticed a sharp memory increase [...]
The Math of Automated Failover
September 18, 2012 By 9 Comments
There are number of people recently blogging about MySQL automated failover, based on production incident which GitHub disclosed. Here is my take on it. When we look at systems providing high availability we can identify 2 cases of system breaking down. First is when the system itself has a bug or limitations which does not [...]
Introducing the pmp-check-mysql-status Nagios Plugin
March 29, 2012 By 2 Comments
Most of the Nagios plugins included with the Percona Monitoring Plugins are purpose-built for specific scenarios that I found in my research to be frequent, severe, or subtle causes of problems in MySQL systems. They are intentionally not generic because I wanted to focus on doing one thing with each plugin, and doing it excellently. [...]
Troubleshooting MySQL Memory Usage
March 21, 2012 By 5 Comments
One of the most painful troubleshooting tasks with MySQL is troubleshooting memory usage. The problem usually starts like this – you have configured MySQL to use reasonable global buffers, such as innodb_buffer_size, key_buffer_size etc, you have reasonable amount of connections but yet MySQL takes much more memory than you would expect, causing swapping or other [...]
How to Monitor MySQL with Percona’s Nagios Plugins
February 22, 2012 By 11 Comments
In this post, I’ll cover the new MySQL monitoring plugins we created for Nagios, and explain their features and intended purpose. I want to add a little context. What problem were we trying to solve with these plugins? Why yet another set of MySQL monitoring plugins? The typical problem with Nagios monitoring (and indeed with [...]
Troubleshooting MySQL Upgrade Performance Regressions
February 15, 2012 By Leave a Comment
So lets say you upgraded from MySQL 5.1 to Percona Server 5.5 and instead of expected performance improvement you see your performance being worse. What should you do ? First if you followed MySQL upgrade best practices such as testing your workload with pt-upgrade the chances of this happening are rather slim. But lets assume [...]
Optimizing InnoDB for creating 30,000 tables (and nothing else)
December 22, 2011 By 5 Comments
Once upon a time, it would have been considered madness to even attempt to create 30,000 tables in InnoDB. That time is now a memory. We have customers with a lot more tables than a mere 30,000. There have historically been no tests for anything near this many tables in the MySQL test suite. So, [...]
Percona Testing: Innodb crash / recovery tests available
December 21, 2011 By Leave a Comment
Not everyone may know this, but there are precious few innodb crash recovery tests available. Some folks have noticed this and asked for something to be done about it, but unfortunately, no tests have been created for the main MySQL branch. The MySQL at Facebook branch has a number of tests that are quite interesting. [...]
Setting up XFS on Hardware RAID — the simple edition
December 16, 2011 By 9 Comments
There are about a gazillion FAQs and HOWTOs out there that talk about XFS configuration, RAID IO alignment, and mount point options. I wanted to try to put some of that information together in a condensed and simplified format that will work for the majority of use cases. This is not meant to cover every [...]
Fishing with dynamite, brought to you by the randgen and dbqp
November 16, 2011 By Leave a Comment
I tend to speak highly of the random query generator as a testing tool and thought I would share a story that shows how it can really shine. At our recent dev team meeting, we spent approximately 30 minutes of hack time to produce test cases for 3 rather hard to duplicate bugs. Of course, [...]
