I am not going to give you a primer  or Tutorial on Tyrant and Cabinet, there are plenty of these out there already.  Instead I want to see what sort of performance we can see compared to MySQL and Memcached, and later on other NoSQL solutions.  Tokyo actually allows you to use several types of databases that are supported, there are hash databases which are very similar to memcached, a table database which is similar to your classic database tables where you can add a where clause and search individual columns, and a ton more “database options”  beyond just those two.  Again my goal is to not make this a Tokyo Tyrant tutorial but rather show one potential role it can play.

More details can be read about here:
http://1978th.net/tokyotyrant/
http://1978th.net/tokyocabinet/

So if we can get performance similar to memcached with Tokoyo Tyrant when using disk based hash tables it would be a compelling replacement for our application here.  It should provide the interface and the same access we saw in memcached but with disk persistence. So let’s look at the numbers:

Tyrant’s disk based hash was almost 2x faster then combining memcached and MySQL, and about 20% slower then the all memory memcached approach.  So for this particular application I would have been much better off not storing my data in MySQL and instead looking outside the database for an answer.  Now sure there are other reasons you may want to keep data in the database… but I am trying to get you to think about your application and if those reasons are really valid.  Helping clients pick the right solution is one of the things we do here at Percona.  If an application requires a database great, but if there is a better solution we want to suggest it.  It’s our goal to make your application perform optimally.

Finally, one concern you have to have is the scalability of your storage solution.  As load, number of threads, and data size increases how does performance differ or change?  One knock on Tokyo -vs- Memcached is Tokyo is not distributed by default.  Now that’s not to say we could not shard it based on a hash, or even build an api with the capability built in ( or use the memcached clients which works! )…  but native support is lacking.  It does support replication which could make some rather interesting architectures in the future.

So lets look at some scalability benchmarks, my server resources are rather limited but I thought I should try throwing more threads and work at the server until it hit its limit and fell over dead.  It’s interesting to see the number of transactions that occur with a given number of threads.  let’s look at some of these:

As expected the smaller buffer pool struggled ( why a smaller buffer pool?  This simulates a much larger data set.  A BP of 256M with 1GB of data, can give similar performance to 20GB of data and a 5GB BP ).  So with 256M BP and 4GB of memcached we were well off the numbers we hit with a 4GB BP+4Gb of memcached ( which is expected ).   Adding more threads even up to 128 threads increased overall throughput but my load average on the server hit 40 and my CPU was pegged.  At 128 threads I was pegging out my CPU across the board.  Also interesting is I started to hit bottlenecks in MySQL/Innodb when I had enough memory but I increased the threads from 64 to 128.  As time permits I should revisit this and look at increased datasets, and look for area’s where Tyrant may stumble a bit.

Bottom line given a specific application and data pattern sometimes a relational database is not the appropriate place for storing data.  A tool like Tokyo Tyrant may not be for everyone or every application, but neither is a relational database.  Before building your next application try and understand whether an RDBMS is really needed or not.

How did I do these tests:

The above number were run with 32 Threads, Tyrant was started with 8 threads and 128M of memory,  memached was started with 16 threads ( 1.4 memcached ), mysql was 5.1 XtraDB.  Each environment had 2 tables each with 2 million rows.  The data was identical. memcached and Tyrant stored a comma delimited string to represent the row.   Mysql was running with 256M allocated to the innodb buffer unless otherwise noted.

What’s next?  Well next I am going to try and continue this series by exploring and benchmarking other NOSQL options and comparing them to database based solutions.  I think showing the performance of a couple of different Tokyo database formats would also be interesting.  What other solutions are people interested in?  I know I have gotten a lot of requests for cassandra #’s, but what else?  Drop a comment and let me know!

Entry posted by matt | 24 comments

Add to: | | | |

At first, <tuning peak performance>. The next, <tuning purge operation> to stabilize performance  and to avoid decreasing performance.

<test condition>

Server:
PowerEdge R900, Four Quad Core E7320 Xeon, 2.13GHz, 32GB Memory, 16X2GB, 667MHz

db_STRESS:
32 sessions, RW=1, dbsize = 1000000, no thinktime

XtraDB: (mysql-5.1.39 + XtraDB-1.0.4-current)
innodb_io_capacity = 4000
innodb_support_xa = false
innodb_file_per_table = true
innodb_buffer_pool_size = 16G
innodb_read_io_threads = 8
innodb_write_io_threads = 8
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 128M
innodb_log_file_size = 512M
innodb_log_files_in_group = 2
innodb_max_dirty_pages_pct = 90
innodb_flush_method = O_DIRECT
(the followings are XtraDB specific general settings)
innodb_ibuf_active_contract = 1
innodb_adaptive_flushing = false
innodb_adaptive_checkpoint = estimate

<tuning peak performance>

At first, tuning the peak performance to use CPU and IO resource more effectively. To avoid mutex/lock contentions are good to use more CPU resource of many CPUs.

This graph shows the peak performance in tps of db_STRESS.

At current settings, “base” in the graph is the perfomance. We can confirm the mutex/lock contention roughly by the SEMAPHORES sction of SHOW INNODB STATUS output.

“xx-lock on RW-latch at 0×7f2ff40a3dc0 created in file dict/dict0dict.c line 1627″

It is index->lock, viewing the source file (and it may be HISTORY table). This is the lock for each index tree. We may be able to disperse the lock using by the partitioning of MySQL. Added the following clause to the HISTORY table definition.

“PARTITION BY HASH(REF_OBJECT) PARTITIONS 16″

Now the performance became to “+partitioned” in the graph. Looking the SEMAPHORES section again,

“has waited at handler/ha_innodb.cc line 7275 for 0.0000 seconds the semaphore:
X-lock on RW-latch at 0xd30320 created in file dict/dict0dict.c line 623″

may be the line which appears for the most times (it is dict_operation_lock). It may be partition specific lock contention. The current XtraDB has the variable to tune the contention.

innodb_stats_update_need_lock = 0 (default 1)

It skip the updating statistics which needs the lock. (it only affects for “Data_free:” value of TABLE STATS). And the performance became “+skip_stats” in the graph.  Then, the next contention at SEMAPHORES section is…

“Mutex at 0×1b3e3e78 created file trx/trx0rseg.c line 167″

may be remarkable (it is rseg->mutex). The mutex is for each rollback segments, so we can increase the rsegs to solve the contention problem. XtraDB can increase the rseg.

innodb_extra_rsegments = 64 (affects to initialization of InnoDB)

Recreated database files with the parameter. Then the performance became “+rsegs64″. At last, the next contention may be “Mutex at 0×28ce8e0 created file srv/srv0srv.c line 982″. It is kernel_mutex, currently we don’t have proper solution for that. The setting seems to be enough for now.

<tuning purge operation>

Next, looking the sequential result in more long term.

The next problem is “History list length” growing to huge size. The value is the number of entries in rollback segment. The entries are used for consistent reading of the older transactions. They can be removed when any transactions doesn’t refer the entry. This removing operation for the entries is called “purge” in InnoDB. The purge operation should be done enough on time, because the huge history list affects to performance.

Basically, the purging is done by master_thread (general background thread of InnoDB). The huge history list makes the purge operation slow, and it interferes  with the other tasks of the master_thread (e.g. flushing dirty blocks, treating insert buffer, etc…). Dimitri implemented a purge_thread to devote to the purging, and also XtraDB has similar purge_thread. Though it seems to make the throughput stabilize, it is not enough still for heavy update workloads. A single purge_thread on one CPU is not enough for updates from user threads on the all of other CPUs.

XtraDB can increase the purge_threads from the next release.

innodb_use_purge_thread = 4

seems to be enough for this workload on the server.

The first graph of followings is sequential throughput [tps] up to 3500 sec.

The next is tracking the “History list length” at the same time.

"Norm 1.0.4": Normal InnoDB Plugin 1.0.4 without XtraDB specific options
"xtra p_t 0": XtraDB 1.0.4-new (no purge_thread)
"xtra p_t 1": XtraDB 1.0.4-new (single purge_thread similar to Dimitri's)
"xtra p_t 4": XtraDB 1.0.4-new (4 purge threads)

The graphs show…

• The purge thread (> 0) helps to stabilize the throughput greatly.
• Increasing the purge threads can suppress the strong growing of the hitory list
• The adaptive checkpoint “estimate” needs the purge_thread… (than the adaptive_flushing does)

And the last 300secs’ average tps are…

"Norm 1.0.4": 5725.47
"xtra p_t 0": 4699.33
"xtra p_t 1": 7130.3
"xtra p_t 4": 9118    (about 60%up from Normal Plugin 1.0.4)

In the end, the faster and more stable performance of db_STRESS benchmark is obtained by these tunings of XtraDB.

—————————————–

(Added 2009.10.29)

<FAQ: Is XtraDB slower than Plugin?>

I’d like to say “no” to this question. We have been adding many tuning options to XtraDB. But they are effective not for all cases, sometimes the performance may get worse because of “not proper” or “too much value”. We should choose the options correctly. XtraDB is based on InnoDB Plugin and we can set XtraDB same to InnoDB Plugin at least. The following graphs are results of XtraDB and Plugin with same options and same database.

top-left (same condition to above graphs):
innodb_flush_log_at_trx_commit = 2
innodb_doublewrite = true

top-right:
innodb_flush_log_at_trx_commit = 1
innodb_doublewrite = true

bottom-left:
innodb_flush_log_at_trx_commit = 2
innodb_doublewrite = false

bottom-right:
innodb_flush_log_at_trx_commit = 1
innodb_doublewrite = false

It seems that XtraDB is not slower than Plugin here at least.

We can start tuning based on these performances using XtraDB specific options!

Why do you make XtraDB slower than Plugin?

Entry posted by Yasufumi | 9 comments

Add to: | | | |

In the case of heavy updating workload, the modified ages of the many blocks are clustered. And to reduce the maximum of the modified ages InnoDB needs to flush many of the blocks in a short time, if these are not flushed at all. Then the flushing storm affect the performance seriously.

We suggested the “adaptive_checkpoint” option of constant flushing to avoid such a flushing storm. And finally, the newest InnoDB Plugin 1.0.4 has the new similar option “adaptive_flushing” as native.

Let’s check the adaptive flushing options at this post.

HOW THEY WORK

< adaptive_checkpoint=reflex (older method)>

< adaptive_checkpoint=estimate (newer method)>

* estimate blocks to flush based on…

• how many modified and unflushed blocks
• progress speed of the transaction log
• the modified age average of all blocks to flush

< adaptive_flushing (default “true” at 1.0.4)>
Its behavior is not based on the oldest modified age %.

* the how many blocks to flush based on…

• how many modified and unflushed blocks
• progress speed of the transaction log
• transaction log capacity

(and adjust along with another flushing)

And it doesn’t exceed the equivalent to “constant flushing (strong)”
of the “adaptive_checkpoint=reflex”

RESULTS

TPC-C like workload (100WH: 16 session full)

innodb_buffer_pool_size = 16G
innodb_max_dirty_pages_pct = 90

innodb_log_file_size = 512M
innodb_log_files_in_group = 2

innodb_io_capacity = 4000
innodb_read_io_threads = 8
innodb_write_io_threads = 8
innodb_flush_method = O_DIRECT

innodb_thread_concurrency = 0
innodb_ibuf_active_contract = 1

<none>
innodb_adaptive_flushing = false
innodb_adaptive_checkpoint = none

<flushing>
innodb_adaptive_flushing = true

<reflex>
innodb_adaptive_flushing = false
innodb_adaptive_checkpoint = reflex

<estimate>
innodb_adaptive_flushing = false
innodb_adaptive_checkpoint = estimate

case1: “innodb_doublewrite = false”

[0~1800 sec.]

[1200~1800 sec.]

averages (1200~1800 sec.)
none:     6868.92
flushing: 6655.92
reflex:     6481
estimate: 6575.88

case2: “innodb_doublewrite = true”

[0~1800 sec.]

[1200~1800 sec.]

averages (1200~1800 sec.)
none:     6569.48
flushing: 5090.12
reflex:     6871.9
estimate: 6609.9

CONSIDERINGS

The new “adaptive_flushing” seems to be tuned for “innodb_doublewrite = false” only.
(cause too much flushing for “innodb_doublewrite = true”)

“innodb_adaptive_checkpoint = reflex” and “adaptive_flushing” need tuning innodb_io_capacity properly.
(The result is based on proper value “innodb_io_capacity = 4000″)

“innodb_adaptive_checkpoint = estimate” is not depend on innodb_io_capacity, and it seems more “soft” result than the other methods.

So,

The “adaptive_flushing” seems good when we use “innodb_doublewrite = false”.

“innodb_adaptive_checkpoint = estimate” may be safe for “innodb_doublewrite = true” for now.

I should adjust “adaptive_flushing” for “innodb_doublewrite = true”.

I have tried to improve the performance. But I couldn’t adjust performance of adaptive_flushing with doublewrite… So, instead of that, estimate is adjusted more.

case1: “innodb_doublewrite = false”

[0~1800 sec.]

case2: “innodb_doublewrite = true”

[0~1800 sec.]

The new “innodb_adaptive_checkpoint = estimate” seems to be more stable than before. The next question is, “Why is estimate + doublewrite better than the other no doublewrite settings?”…

Also tested in another configuration to check the behaviors in another balance of IO bound.

<changes>
innodb_buffer_pool_size = 2G   (1/8)
innodb_log_file_size = 128M    (1/4)

case1: “innodb_doublewrite = false”

[0~1200 sec.]

case2: “innodb_doublewrite = true”

[0~1200 sec.]

“innodb_adaptive_checkpoint = estimate” seems stable but seems to flush much in its initial phase. The adaptive_flushing seems to be more soft in this cases.

Hmmm….

We should choose suitable adaptive_xxx for each workload for now…

Entry posted by Yasufumi | 12 comments

Add to: | | | |

The Brian’s portion of webinar will focus on the Schooner offering as example of flash based appliance – one of solutions I mention in my presentation.

It should be interesting whenever you’re interested in Schooner appliance offerings, flash or scaling MySQL in General.

You can follow this url to register.

Entry posted by peter | 2 comments

Add to: | | | |

Let's start with this query, which returns zero rows but does a full table scan. EXPLAIN says there are no possible_keys.

Don't try to figure out the meaning of the query, because that'll add complexity to the example In the simplest case, we want to put the most selective column first in the index, so that the number of possible matching rows is the smallest, i.e. we find the rows as quickly as possible. Assuming that all the columns have an even distribution of values, we can just count the number of matching rows for each criterion.

This is pretty simple -- all I did was wrap each clause in a SUM() function, which in MySQL is equivalent to COUNT(number_of_times_this_is_true). It looks like the most selective criterion is "status=waiting". Let's put that column first in the index. Now, pull it out of the SELECT list and put it into the WHERE clause, and run the query again to get numbers within the subset of rows that match:

So we're down to a reasonable number of rows (the count() is changing because I'm running this on live data, by the way). It looks like the 'source' is no more selective, that is, it won't filter out any more rows within this set. So adding it to the index would not be useful. We can filter this set further by either the 'no_send_before' or the 'tries' column. Doing so on either will reduce the count of matches for the other to zero:

That means we can add an index on either of (status,tries) or (status,no_send_before) and we will find the zero rows pretty efficiently. Which is better depends on what this table is really used for, which is a question I'm avoiding.

Entry posted by Baron Schwartz | 13 comments

Add to: | | | |

The most popular equation, of course, is the one in the error log after a crash: "It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections". This was never true -- not even when ISAM and friends were the only storage engines, which was before most of us remember. And at a minimum, most servers these days need to consider the added memory for InnoDB: specifically, the buffer pool. (There's also an innodb_additional_mem_pool_size but it's typically set pretty small, like 20M).

But even that's not enough. Staying with InnoDB, we also need to count the data dictionary. This can take many gigabytes on servers with a lot of tables. We're trying to fix that, but it's unbounded in a non-Percona server. And InnoDB has other needs that it doesn't actually promise to fit within innodb_additional_mem_pool_size, by the way. If it needs more it will allocate it. innodb_additional_mem_pool_size is just a handy way to avoid repeated malloc() calls by doing it all up front in one call.

And what about the query cache and the table cache and the size needed for each thread (thread stack, etc)? Those take memory too. The query cache can use quite a bit if it's configured to. There are a bunch more global buffers to think about; that's just a partial list. And there are some "global buffers" that aren't really global. Suppose you set sort_buffer_size to 128K -- I can quite well connect and say "set @@sort_buffer_size := 1 * 1024 * 1024 * 1024;" and my sort_buffer_size is now 1GB. And a single query may cause multiple sort buffers to be allocated. You can't control that or calculate it at a global level.

But let's keep going! Any stored code (triggers, stored routines, etc) uses memory to execute; so do temporary tables. The following query creates at least 2 temporary tables:

You can have many of those per query. Sorting and grouping and DISTINCT and UNION may/will also create temp tables; the same applies to views and probably some other things I'm forgetting.

Let's not forget about the memory needed to parse and optimize and execute queries; mysqld creates internal structures for the query plan, for example, and keeps it around until the query is done. And there are prepared statements, which you can create and never deallocate.

Any more? Of course... I can also set a bunch of user variables -- they use memory too. And they can be really big (but no bigger than max_allowed_packet). And keys for user locks! I set my max_allowed_packet bigger and ran the following on my laptop:

Now I've got a 100M user variable and a 100M lock key. Note how long it took to run these statements! And now mysqld is suddenly using 324M of memory, up from 124M.

I'm sure you can think of other things I've forgotten here (the manual page has a bunch). My point is there simply is no formula to compute the maximum possible memory usage, and there cannot be because there are some things that have no defined limits.

It's not something that's worth worrying about much, in my opinion. I think it's far better to use a monitoring tool to watch the actual memory usage over time and see how it varies. When I'm initially setting up a my.cnf for a server, I usually calculate the Big Ones like the buffer pool, query cache etc; leave some memory for other needs; and see what happens. The results are workload-specific.

Entry posted by Baron Schwartz | 16 comments

Add to: | | | |

My initial idea was - I'll just analyze queries in the slow query log generated by our mysql build running with long_query_time=0, get the slowest ones and work on them. It wasn't really all the way I expected..

For a showcase I have created a function "whatstheweatherlike". Let's call it and see what shows up in the slow query log:

Not overly useful for query optimization, is it? Well, I can still aggregate the slow query log with mk-log-parser and find routines that MySQL spends most of the time on, but I don't want to go over the complicated logic of stored procedure just to figure out what queries are being executed. This is where Jeremy's SHOW PROFILE patch - which is included in all Community tree based Percona MySQL builds since it is part of MySQL community version ever since 5.0.37 (note - not all Percona builds are done on Community tree) - comes into play. Let me show you how this works:

So now not only do we get to see what queres were executed, how much time they spent executing but we can also get a profile for each of the queries individually. I think this is pretty cool and it helped me a lot. Sure, you may still need to check routine and track the logic so having constants that were used rather than variables would be even more useful. Nevertheless I think this is a great way to find what's happening under the scenes so you could start digging deeper.

On the other hand, this way of manual profiling you can't really see real time stats, for example if it was slow because of locks, in the logs you may not see it. I will ask my colleagues to check if it would be hard to implement configurable variable for switching between routine and query logging, so maybe one day we'll be able to have both ways of looking at it.

Entry posted by Aurimas Mikalauskas | 3 comments

Add to: | | | |

The time is critical for many performance optimization tasks not only because of labor associated expenses but also because of the suffering - slow web site means your marketing budget is wasted, customer not completing purchases, users are leaving to competitors, all of this making the time truly critical matter.

So what can be the goal ? Generally I see there are 2 types of goals seen in practice. One is capacity goal this is when the system is generally overloaded so everything is slow, when you're just looking to see how you can get most out of your existing system, looking for consolidation or saving on infrastructure cost. If this is the goal you can perform general system performance evaluation and just fix the stuff which causes the most load on the system. MySQL Log analyzes with Mk-Log-Parser is a very good start for a ways to generally optimize MySQL load on the system.

Latency Goal is another breed. The system may not look loaded but some pages still may want to be loading much slower than you like. These goals are not system wise but they are much more specific to the different user interactions or even types of users. For example you may define goal also "Search pages have to have response time below 1 second in 95% cases and below 3 seconds in 99% cases". Note We're specific to the user interaction - people are used to Search taking longer time than other interactions for many applications, and also we speak about percentile response time rather than "all queries". It is surely good all search queries complete in one seconds but it is too not practical. The goal description may be more specific too - for example you may have different response time guidelines for pages which are requested for real humans vs search engine bots (which are often quite different in their access pattern) or you may define "large users" as users having more than 100.000 images uploaded and measure the response time for them specifically because this group has its own performance challenges.

Looking at Latency it is also much more practical to look from the top of the stack. If you look at MySQL log you may find some queries which are slow but it is hard to go back from them to what is really important for the user and so the business - the page response times. Furthermore. It is not enough in many cases to focus only on Server Side optimization - the Client Side Optimization is also quite important in particular for aggressive performance goals and fast back-end. This is why we added this service to Percona offerings.

If Server side or Client Side performance optimization is going to be more important for your application depends on the application performance a lot. The better your application is the more Client Side optimization you will need. For example if it takes you 30 seconds to generate the search results and 3 more seconds to load all style sheets images and render the page server side optimization is more important. If you have optimized things and now HTML takes 0.5 seconds to generates an extra 3 seconds become the main response time contributer which has the highest performance optimization potential.

But let us get back to the Server Side Optimization. Lets assume our performance goal applies to the HTML generation rather than full page load on the client. So meet our goal we should look at the pages which do not meet our goal, which is pages which take more than 1 second to generate in given example.

For goal driven performance optimization it is important there is enough instrumentation and production performance logging in place so you really can focus on hard data in your work. For small and medium size applications you can log all requests to MySQL table for larger ones you can log only small portion of them. I usually keep one table per day so it is easy to copy the data to a different box for data crunching and remove the old ones.

The log table should contain URL, IP and all the data you need to be able to repeat request if you need to. It may include cookie data, post data, logged in user information etc. But the real thing is number of times which are stored for request. wall clock time - is the real time it took to generate the page by server backend. CPU Time This is the CPU time needed to generate request (you can split it to user and system if you want) and when there come various wait times - mysql, memcache, sphinx, web services etc.

For web applications doing processing in a single thread the following simple formula applies wall_time=cpu_time+sum(wait_time)+lost_time The lost time is the time which was lost for some reason - some waits we did not profile or waits we do not have control of, for example when processing had to wait for CPU available to do processing. For multi-thread application it is a bit more complicated but you still can analyze critical path.

If you have such profiling in place all you have to do is to run the query to see what are contributing factors to the response time of the problematic pages:

Why looking only at such pages is important ? This is because if you look at all pages rather than problematic subset it may lead you away from your goal. For example it is very possible among all pages we would see CPU usage as the main factor because sphinx and MySQL respond from cache.

We however see for pages which have the problem it is Sphinx which accounts for most of the time.

Looking at the data such way we have two great benefits. First we really understand what is the bottleneck. Second we know what performance gain potential is. For example in this case we could spend a lot of time optimizing PHP code but because it takes only 10% of response time in average even speeding it up 10 times we would not get more than 10% response time reduction. At the same time if we find a way to speed up Sphinx we can reduce response time to its half.

Note in this case there is some 16% of response time which is not accounted for. Large portion probably comes from memcache accesses which are not instrumented for this application. In this case this portion is not the biggest part yet but if we'd speed up Sphinx and MySQL dramatically we would have to go and look into better instrumentation so we can look inside this black box.

Once we know it is Sphinx which causes the problem we have to go and find what queries exactly are causing it - this can be done by adding request ID as comment to Sphinx log so you can profile it carefully or you can add tracing functionality to the application. All the same. Once you found the queries causing the problem you see the ones which cause the most impact and focus on optimizing them.

There are multiple ways to optimize something, my checklist is usually get rid of it, cache it, tune it, get more hardware in this order. It is often it is possible to get rid of some queries, cache them, tune them so they are faster (often at the same time changing semantics a bit) and if nothing helps or can be done quickly we can buy more hardware, assuming application can use it.

Once you've performed optimizations you can repeat analyzes again to see if performance goals are met and where is the bottleneck this time.

As a side note I should mention looking at performance statistics for the day overall is often not enough. Application performs as good as it performs during its worst times so it is very good to plot some graph over time. Sometimes an hour base may be enough but for large scale application I'd recommend to looking down to 5 minutes or even 1 minute intervals and making sure there are no hiccups.

Check the stats from the application above for example:

As you can see in this case during certain hours the average type of bad queries skyrockets and it becomes 90% or so driven by Sphinx. This tells us there is some irregular activity (cron jobs?) is happening and it affects Sphinx layer significantly.

Such goal based from top to bottom approach is especially helpful for complex applications using mutliple components (like sphinx and MySQL) or multiple MySQL Servers because in these cases you often can't easily guess the component which needs attention. Though even for less complicated single MySQL server application there is often the question if it is MySQL server causing the problem or if application code needs to be optimized.

Entry posted by peter | No comment

Add to: | | | |

A full performance audit is far more extensive than this article can cover, and might wander into Apache, networking config, caching layers, etc. Wherever the problem is, we'll track it down. I won't talk about that. That's not because I want to keep secrets from you. To the contrary, I'd love to share it all with you. But that's a huge job; it will take many pages, and I'm not going to write that much.

The kickoff call

There's actually a step before the performance audit begins. We call this a kickoff call. We get on the phone with the key technical staff on the client's side and discuss the application in general. If possible, we take a brief look at the server beforehand, so we can ask more intelligent questions and skip obvious things. This call is sometimes up to an hour long, if we're discussing a lot of things like how to build for massive scalability, how to do read-write splitting without breaking the user experience, how to take online non-blocking backups, or things like that.

But in general it can be a lot shorter than clients expect, because

• time is money
• your application is not as unique as you think, and we already know a lot about it by browsing your site or hearing you say "we are a SaaS hosted application for data mining stock market transactions," or whatever the case may be
• we've generally seen just about everything people throw at databases

Many people assume their application is difficult or somehow different, and that we need all kinds of schema diagrams and code listings, but the truth is people often have the same insights into particular problems, and therefore they try similar solutions. And besides, most applications have a lot of the same components. Tagging, friends, queues, click tracking, search, paginated displays -- we've seen these and dozens of other common patterns done a hundred different ways.

The kickoff call has some other important goals too. We want to know what your concerns are at this stage. Is it the current performance, future performance, scalability, ability to recover from disasters? We also want to know what the operating parameters are. Can we suggest upgrades, rewriting queries, or whatnot? Sometimes there are rigid constraints on what types of solutions we can propose, and we need to know ahead of time so we don't spend time on things you have no way to change. An example of this is a third-party product whose code you are unwilling or unable to change.

We also want to know the operating mode you expect us to work in. Maybe you want us to analyze and present recommendations for your consideration, and take a look-don't-touch approach. Or, maybe you just want us to fix things and tell you later what we did. Either is a completely valid approach. You tell us what you want, and that's what we'll do (of course, we will also tell you what we think is best for you, which is our job).

After the kickoff call, we write some notes, and then get down to the audit itself. We usually have two people on the kickoff call so that we don't have a single person knowing everything about you (only one is billable, though). The second person will be from the same team within Percona. However, the audit itself is generally a single person, with another person reading all the notes and generally keeping in touch with what the lead consultant does. We want two sets of eyes on things whenever possible. This also has value in case you decide to continue on to a longer relationship with Percona, which is reasonably common; in that case you will establish a solid relationship with the team, and group involvement early on is a good start to this.

What to look for during the audit

During the audit I look for outliers and anomalies in every aspect of the server's configuration and performance and contents. I'm looking for characteristics that are much bigger or smaller than normal, or things I don't usually see. Anything out of the ordinary or out of proportion. These things are often relatively easy to catch if you have enough experience. It's a little hard to describe without making it sound like it's some secret handshake, but it's really not magical. You're either familiar with what a server normally looks like, or you're not. It's just like a mechanic who listens to the car and says "I hear the valves knocking."

In other cases there may be things that are NOT easy to catch, and may require a lot of experience and investigation. The work we do is often not simple at all. It's impossible to predict. A customer may call and ask "how long to debug server lockups?" It could be ten minutes, or it could be a really major effort chasing down something like a bug in a hardware component, and you never know ahead of time.

In either case, the point to know here is that we can work with systems that are at any level of tuning, from completely untuned to a system that's already had a lot of expert attention. I use "tuning" in a generic way here -- we focus on far more than my.cnf files. In fact my.cnf files are usually one of the smallest levers we can pull for server optimization. Schema and query optimization can give much greater improvements, for example.

If you're familiar with Malcolm Gladwell's excellent book Blink, you know the term "thin-slicing." That describes an experienced consultant's approach to the problem very well.

The audit itself

OK, so here's a very basic audit, whose scope I'll keep within just a MySQL server. The first step is to log into the machine via SSH and open up a text editor (gedit, notepad, what have you.) I want to keep this open and paste everything I see into it. If there's no record later that someone else can follow, I am not doing my job.

I would also note that the commands I'll show here are the lowest common denominator. We often use tools to gather a lot of the data for us and speed the process, when possible, but the core commands I'll show are what we can always fall back to if that's not possible.

Gathering information about the server

I start off with these commands:

• date
• uname -a
• cat /etc/*release

The first command is invaluable for later reference. The others will show exactly what kind of server I'm looking at, assuming I'm on a GNU/Linux box (I have to adapt to whatever environment I'm in -- obviously a FreeBSD box is different). For example,

From this, I can see a couple of important things. One thing is that I'm looking at a 64-bit OS, which is important to know. I also know I'm on RHEL and I know the kernel -- knowing this may tell us a lot of useful things about the system.

Next I continue looking around the server. Holding with the assumption of GNU/Linux, I keep working to get a picture of the OS and hardware.

What I see here is that this is an 8-core machine, two quad-core processors at 2GHz. I also know some things about the CPU, such as the model number and the cache size, etc. What else is important is the 'lm' flag, which tells me this is a 64-bit CPU. From this and what I saw earlier I can see whether someone installed a 32-bit OS on 64-bit hardware, which is important if you want to allocate a lot of memory to mysqld.

Next I look at a couple of all-in-one commands. For example, 'top -n 1' and 'uptime'. That shows me what processes are running, but it also shows me load average, memory sizes, and a bunch of other stuff. In this case I'm on a machine with 32GB of memory and there is very little load. I see that mysqld is using 10g of memory with 17g virtual size, and there's practically nothing else running (this is a dedicated master server). The machine has a lot of swap, but basically none is used. This isn't all that relevant, as you'll see later, but in some cases it might be.

I continue my investigation into the machine itself with these commands:

• mount
• df -h
• dmesg
• pvs; vgs; lvs

These commands show me what filesystems are mounted, how much space they have on them, and what kind of I/O system is under the box in general. dmesg is useful for several reasons. For one thing, it has a lot of information on the box's hardware, including RAID controllers, installed memory, and so on in the boot output. In addition, at the tail it has information on core dumps and so on. It's possible that it has filled up with stuff and the boot output has been lost, but in many cases there is a lot of information here that's hard to get elsewhere.

If I see an LSI MegaRAID card, which is pretty common, I can check whether MegaCli is installed, and use it to get information about the physical drives installed; then I can find out exactly what they are, both in terms of size and spindle speed. I need to know how many physical spindles are under the box, and how fast they are. I can also get information about the all-important battery-backed write cache, how it's configured and what its status is. I can see the drive status, too. Sometimes a failed drive is present but no one has noticed it yet! (By the way, we can also help you install and configure monitoring and graphing systems, if you don't have any. These will help catch such problems.) If MegaCLI isn't installed there may be other ways to get this information, too.

The last three commands are for giving me information about LVM devices, volume groups, and logical volumes. At this point, unless I've seen something that I want to investigate further, I have a pretty good idea what hardware and operating system I'm working with.

System performance

Next I investigate what the system is doing, in terms of performance. There are two key commands for this on most operating systems (there are others, such as mpstat or dstat, that I may also elect to use in some cases):

• vmstat 5 5
• iostat -dx 5 5

If you don't know how to read these, there's a full explanation in our book, High Performance MySQL 2nd Edition (it's not in the first edition). From this I get an idea where any possible problems might be: is the disk saturated? Is the machine swapping actively? (This is more important than whether it's using swap.) There are a lot of combinations of possible things you can see here, so I will not try to explain it all. Basically you need to know everything there is to know about these two commands and their output.

Based on what I've seen so far, I may look at other things, such as ifconfig (which shows good stuff like dropped packets).

Starting to look at MySQL

Assuming I don't see anything noteworthy here, I'll move on to the MySQL server. I mean, let's just assume the physical hardware and the machine setup is fine, and the client has asked me to figure out why the website is slow; assume also I've checked out Apache and network config, etc, and I don't see anything wrong there. So at this point, it looks like MySQL is probably a good place to look for performance problems. Everything I've done up till now is just due diligence; I've gotten my bearings on the server as a whole, and now I'm ready to see what's up with the database.

The first step is to find out what MySQL instances are on the machine. It doesn't do to just assume there's a single instance in /var/lib/mysql and it's reading from /etc/my.cnf. I've seen a lot of cases where there's a /var/lib/mysql and an /etc/my.cnf, and the server is installed in /customsoftware/mysql with a different my.cnf file. The way to find this out is to look at ps:

In this case, yes it's all default. It's listening on the usual port, etc etc. Nothing unusual. Next I check what the MySQL version is, and I check (with the 'file' command) whether /usr/libexec/mysqld is a 64-bit build, if I'm on a 64-bit OS.

Examining MySQL configuration

Then I'll look at my.cnf quickly for anything weird:

My goal here is to look at the my.cnf without seeing all the comments, which a) are often a small novel, b) often say what the sysadmin thinks s/he's doing, which might not be what is really being done. I want to look at it without polluting my brain with that. I'll just scan this for anything odd. In contrast to what you might think, spending a lot of time here isn't usually a good idea, because what's left out is often more important than what's included. For example, the absence of skip-name-resolve is easy to miss, as is the absence of innodb_log_file_size.

I won't give you a full rundown over every option in the file. But again, you learn quickly if something strange is here.

This whole time I've been copying the output of the commands and saving them in a text file. I'll attach this text file to the issue in our CRM system so I (or someone else) can look at it later, see what I saw, and follow my line of reasoning about any suggestions I make.

The next thing to do is grab 'mysqladmin variables' and put it in the text file. I don't spend a lot of time looking at this. For one thing, all the values are in big units, so it's hard to read. If the InnoDB buffer pool is set to 22GB, it's much easier to see 22GB than 23622320128, especially since that number is buried next to a lot of other long numbers. They make my eyes hurt, and there are better ways to do this (I will run mysqlreport to get friendly numbers I can read easily). But what I will do is scan the output for something strange that might be caused by a syntax error in the my.cnf file. For an example, take a look at this bug report (not a bug).

Looking at MySQL status

Now we're starting to get into the really interesting bits. Alas, these are the bits where there is the most variability, so I'll have to be even less detailed and a little vague. The next command is this:

I let that run for at least two iterations. The first iteration is the current values since the server was booted; the second and subsequent are incremental differences. I'll usually capture two of these. It looks like this:

Then I format the second set of values up beside the first set for ease of reference (an easy task with an automated too, or a quick Vim command if I'm doing this the manual way.) The result looks like this:

This output is very useful when writing the report to the client. Now I go through it line by line looking for things that look wrong. When I find something, I compare it against the server's variables or whatever other data I need, and write something instructional to the client about it.

Preparing to report findings

I should mention here a bit about how I write the report to the client. I copy and paste the interesting snippets from what I've been gathering, save it in a text file, and explain what I'm seeing to the client. Anything I see that looks wrong should be explained in detail, because my goal is to teach the client, not just to awe them with my knowledge and get "I could never understand, O Almighty Percona, here's your money" in reply. If the client doesn't understand what I'm doing, I've done something wrong. Clients need to understand so that they know they're getting good service. A Percona consultant never says "just trust me" unless the client really insists "I don't want to know."

That's also why I'll comment on things that seem good. Clients need to know that I'm being thorough. Once upon a time, a Percona consultant was called back in to do some further work about 6 months after Peter helped a client. Now, if Peter worked with the client 6 months ago, you have big shoes to fill, and getting further gains will probably not be easy. So this consultant looked through things, spent about 2 hours, and came back to the client with "there's not a lot of performance left to gain here. You can do X, Y and Z, and that's about all I see. Beyond that, we need to talk about making your overall architecture more scalable." And the client said "that's all? Didn't you do any work? Peter's report 6 months ago was a small encyclopedia!" The mistake here was that the consultant didn't show all the things he'd checked, which was extensive indeed. After a quick explanation the customer understood that the consultant had really done a lot of work, but it's better to show the client the full extent of the investigations in the first place, and not just show "here's the 3 bad things I found in 2 hours." We also make sure that we refer back to the previous issue's findings, so that we don't duplicate efforts from the earlier optimization.

Back to the output of mysqladmin. When I send the report to the client, it typically looks something like this. First, I explain the meaning of the output, so it's clear what I'm showing them. Then, I go through it line by line, like this:

+-----------------------------------+----------------------+----------------------+
| Variable_name                     | Value                | Value                |
+-----------------------------------+----------------------+----------------------+
| Aborted_clients                   | 205174               | 6                    |

Here we can see that something is disconnecting ungracefully
(without closing the connection properly).  This is happening 6
times every ten seconds.  We need to figure out what is happening
because it can indicate something else is wrong.  This is not a
big deal for performance, it's just something to figure out.

... snip ...

| Com_admin_commands                | 255868807            | 11893                |

Why is something running an admin command constantly?  This
is generally a Ping command, which is useless.  Something is
pinging the server about 1200 times per second.  Often this is
from some ORM system, such as Class::DBI, which does $dbh->ping
before each and every query.

Why is Ping useless?  Well, it is intended to tell you whether the
server is alive.  The trouble is, if it succeeds, it tells you nothing.
An instant later, when the ORM system actually runs the query,
the server could have died.  The ORM should just be running the
query, and if the query fails, then you know from the error code
that the server is dead.  So a successful Ping is meaningless.

On the other hand, pinging can be a serious performance problem.
Not really for the server, since it's a cheap query.  It adds hardly
any load.  But for the application, it's another network round trip.
It is a "garbage" query that just adds latency and makes the overall
time to complete an action longer.  If your application runs very short
queries, this can be a non-trivial part of the app's response time.

And so on, and so on. I'll often refer to other things; for example, when I'm examining the status values for Created_tmp_tables and Created_disk_tmp_tables, I'll paste in

If these are not the same size, as often happens, I'll explain that the minimum of the two is used to determine when an in-memory temporary table will be converted to an on-disk table, which is an expensive operation.

In this way, I work through the status and the variables, and explain to the client what I can deduce about the server from them.

I would point out that I don't spend a lot of time on settings and status. It is not where the greatest gains are to be found. The real gold mine is yet to come. After you've done it a few times, you can go through the settings and status pretty quickly.

Analyzing MySQL's workload and data

After I'm done with that, I'll grab a few snapshots of SHOW FULL PROCESSLIST, and look for odd values in this. For example, if I see a lot of queries in odd statuses, like "statistics", I know something weird is going on, and I look deeper. I also look at the queries themselves. If the client has long-running queries, it can be pretty easy to catch. Maybe they're long-running because they're in Locked status, which might be an indication that it's time to convert MyISAM tables to InnoDB. (But then again, it might not be.) Or maybe the client is doing queries like "... WHERE client IN (SELECT id FROM clients)" which is a really bad query plan.

After this I'll look at SHOW INNODB STATUS\G and see if there's anything worth commenting on there. Depending on the workload, there might be a lot of substance here. There's also a very thorough section on this in our book.

Next I look at the data in the server and see what I think about it. If the server is not heavily loaded, I may even do some INFORMATION_SCHEMA queries to help me find the biggest tables and so forth. If the server is heavily loaded or has a lot of data, touching the INFORMATION_SCHEMA can take a barely-running site and bring it down, so I do this with great care! If it's not appropriate I may run 'du' on the data files, or simply skip this step. This information can help me determine what kinds of things I ought to look for later during query analysis, and also might show me things like huge tables that should be archived, tables that I should be aware of if I see them in queries, or what have you.

I will also look at the error log. The error log shows all sorts of interesting things. Basically, anything but silence in the error log is interesting and needs to be investigated. You might find that there are InnoDB tables whose .frm files exist but have been dropped from InnoDB, for example. (The INFORMATION_SCHEMA query will surely trigger errors about this.)

There are a bunch of other things I'll look at, including checking for default users in the mysql.* tables, running mk-duplicate-key-checker to find redundant indexes, and so on. As with settings and status, these set the stage for bigger gains to come later, so I don't spend all that much time here, although there is occasionally something really bad that can be fixed and bring big gains.

Schema, query, and index optimization

Analyzing and optimizing a server's table and index structures, and the queries that run against them, is by far the most important thing to concentrate on. All the other work is a prerequisite to understanding the context within which these things operate. For example, if the workload is a star schema data warehouse, it is important to know a lot about the I/O subsystem. And by now I know that I should concentrate in certain areas -- there may be hundreds of tables, but at this point I should have identified a handful of tables that really matter. That's why I save this part of the analysis for last.

You cannot consider any one of these factors (schema, queries, indexing) in isolation, because they are tightly bound together, and tweaking one will often have effects on the others. Each decision is a consideration of the costs and benefits on the system as a whole.

The queries the system runs, and hence the slow query log, is one of the more fruitful ways to analyze the system. What I usually do is ask the client to enable it a day before the audit if possible, and set it to one second. That way there's at least one "cycle" of queries in it. Then I run the following:

By default, this outputs the top 10 most expensive queries, in total execution time. By the way, the slow query logs in the stock MySQL server are extremely limited, and for serious analysis it's basically mandatory to use a server that has the Percona patches for microsecond logging and additional information in the slow query log output. The stock MySQL server's limitation of one-second granularity makes it hide problem queries that are faster than one second (which in a high-performance system is virtually every query.) If it's not possible to use a patched binary, we can use MySQL Proxy, packet sniffing, or other techniques to get more information than the slow log gives us; but by far the richest and most efficient source of analysis information is our custom server, which we have designed exactly for the purpose of giving us as much insight as possible into the server's true workload. At the moment there is no other technique that approaches the amount of information a Percona build can give you.

Having found the desired information about the queries by any means necessary, I find out which are the most likely to give the greatest gains. I use a dual approach for this: I look for both the queries which cause the greatest load on the server (in aggregate) and which cause the greatest latency for the application. Again, it's a process of finding outliers by some criteria. Both types of queries matter. Some may be extremely fast, but run very often; others may be run seldom but take a long time to finish. And it's important to focus on ones that can give the greatest improvement for the customer. I can often tell at a glance whether a query is going to be possible to improve a lot, and a simple mental calculation can then tell me how much total gain I can get from it. Queries are not just "bad" or "good" -- it's a question of where they fall on the scale.

Now I analyze these one at a time. For each query, I run EXPLAIN if possible (rewriting non-SELECT queries if needed, and using careful judgment for queries with subqueries in the FROM clause, which will actually execute the inner subquery!), and examine the query plan. This is where you need to really know how to write queries and how EXPLAIN works. At Percona, we have peer training constantly on our internal IRC channel and mailing list. We share all sorts of dirty tricks and neat ideas with each other. Nothing is off the table. Queries can be rewritten. Indexes and data types and table structures can be changed. Queries can be broken into pieces, combined, or even eliminated entirely (we may suggest caching, or tell the customer to evaluate Sphinx, or something like that.)

In the common case, though, a query simply needs a rewrite or a new index or something. In this case we show the query and the information mysqlsla outputs about it (execution times, etc), the EXPLAIN plan, and the desired modifications to the query or the table. We explain how to interpret what we're showing, and why the proposed modifications are better. If the client approves it, we may make a copy of the table, make the proposed modifications to it, and show the difference afterwards. Or we might have a test server to run on. It varies widely; some clients have a test server, some don't; some have a formal QA process; some don't. It is very customer-specific, and we work with what we have.

Sending the report to the client

After the slow query log analysis is done, I format all my analysis for sending, add in things I may have noticed along the way (comments on backups, for example) read over it once again and make sure I didn't miss anything important or write something in a confusing way, and then send it to the client. Then I usually call to discuss, or just confirm that it was received and wait for a reply. In most cases the client has a lot of work to do -- sometimes weeks of application changes. Afterwards there's usually more to be gained by doing another pass through.

We may also want to have another call with the customer and talk about changes that can or should be made at a much higher level. If we see that the application's overall architecture needs to be changed, that's something to discuss. Customers usually want us to validate their application's overall scalability and whether it'll grow to meet their demands for some period of time. We also frequently bring the discussion to topics such as monitoring, alerting, backups, caching, reverse proxies, and high availability solutions, all of which we can frequently help customers set up much more efficiently and with better results, due to our knowledge of what works in the real world and where there's quicksand to be avoided.

The changes Percona's performance audits typically suggest can be really significant, and can completely change the performance profile and workload of a server, so it's often worth another iteration, and sometimes even more after that. A really thorough audit can take up to ten hours! The initial round usually takes less than two, however. The ten-hour cases are usually for really complex applications, or apps that have a lot of stored procedures, which are more difficult to analyze because you have to pull out the statements and analyze them individually. The goal is always to develop X-Ray Vision and see through the symptoms to the underlying causes in every aspect of application performance.

Customer interaction

In all cases, the customer's guidance is our first rule. The customer determines the depth and complexity of the analysis we do. A lot of times, customers will tell us up front to limit the work to some number of hours, such as 3 hours. Knowing this ahead of time gives us a framework within which we have to deliver the maximum value to the customer. If we have less time, we fix the "sore thumbs" and make notes of places where more effort is likely to pay off, and mention those to the customer. Or we economize by giving the customer easy choices so they can decide which things to have us fix, which to fix themselves, and which to just ignore.

The above process may look like a static recipe that you can just mechanically execute, and in some cases that's truly what we do. However, we don't just go through the motions. The "recipe" above is only a framework within which we work. Knowing the rules of a performance audit, and why they are the way they are, means you can know when and how to break them for the client's benefit. It's like knowing openings in chess -- the great chess players study openings exhaustively, but they don't robotically play them in tournaments.

For example, although it may sound as though I go work for a while and then emerge with a report, it's not that way a lot of times. Flexibility and exercising judgment about when it's appropriate to interact with the customer are a hallmark of our approach. If we need to be interactive and keep in touch with the client early and often, we do that. A lot of times I'll start out with a quick look and have a second brief call with the customer, or send a quick email and ask for feedback on the direction I'm going; or just get on an instant messaging chat and tell the customer "here's the top thing you can start working on while I keep investigating." We make our process as interactive and flexible as the situation calls for. Early, frequent feedback lets us make small adjustments to the direction we're going, communicate to the client how much time we're spending, and shortcut effort if it's not useful, e.g. skipping analysis for a query the client's developers are in the process of removing from the application completely.

That's all! I hope this has been an informative and fun trip through the world of a typical Percona performance audit, although as you've seen there really isn't any such thing. Post your questions in the comments, and I'll try to answer them as time permits.

Entry posted by Baron Schwartz | 14 comments

Add to: | | | |

The basic point is that your log file needs to be big enough to let InnoDB optimize its I/O, but not so big that recovery takes a long time.  That much Peter covered really well.  But how do you choose that size? I'll show you a rule of thumb that works pretty well.

In most cases, when people give you a formula for choosing a configuration setting, you should look at it with skepticism.  But in this case you can calculate a reasonable value, believe it or not.  Run these queries at your server's peak usage time:

Notice the log sequence number. That's the total number of bytes written to the transaction log. So, now you can see how many MB have been written to the log in one minute. (The technique I showed here works on all versions of MySQL. In 5.0 and newer, you can just watch Innodb_os_log_written from SHOW GLOBAL STATUS, too.)

As a rough rule of thumb, you can make the log big enough that it can hold at most an hour or so of logs. That's generally plenty of data for InnoDB to work with; an hour's worth is more than enough so that it can reorder the writes to use sequential I/O during the flushing and checkpointing process. At this rate, this server could use about 110 MB of logs, total. Round it up to 128 for good measure. Since there are two log files by default, divide that in half, and now you can set

Does that look surprisingly small? It might. I commonly see log file sizes in the gigabyte ranges. But that's generally a mistake. The server I used for the measurements above is a big one doing a lot of work, not a toy. Log file sizes can't be left at the default 5MB for any real workload, but they often don't need to be as big as you might think, either.

If this rule-of-thumb calculation ends up showing you that your log file size ought to be many gigabytes, well, you have a more active write workload. Perhaps you're inserting a lot of big rows or something. In this case you might want to make the log smaller so you don't end up with GB of logs. But also realize this: the recovery time depends not only on the total log file size, but the number of entries in it. If you're writing huge entries to the log, fewer log entries will fit into a given log file size, which will generally make recovery faster than you might expect with a big log.

However, most of the time when I run this calculation, I end up finding that the log file size needs to be a lot smaller than it's configured to be. In part that's because InnoDB's log entries are very compact. The other reason is that the common advice to size the logs as a fraction of the buffer pool size is just wrong.

One final note: huge buffer pools or really unusual workloads may require bigger (or smaller!) log sizes. This is where formulas break down and judgment and experience are needed. But this "rule of thumb" is generally a good sane place to start.

Entry posted by Baron Schwartz | 25 comments

Add to: | | | |