<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>MySQL Performance Blog &#187; tips</title>
	<atom:link href="http://www.mysqlperformanceblog.com/category/tips/feed/" rel="self" type="application/rss+xml" />
	<link>http://www.mysqlperformanceblog.com</link>
	<description>Everything about MySQL Performance</description>
	<lastBuildDate>Sat, 21 Nov 2009 03:11:18 +0000</lastBuildDate>
	<generator>http://wordpress.org/?v=2.8.4</generator>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
			<item>
		<title>&#8220;Shard early, shard often&#8221;</title>
		<link>http://www.mysqlperformanceblog.com/2009/11/16/shard-early-shard-often/</link>
		<comments>http://www.mysqlperformanceblog.com/2009/11/16/shard-early-shard-often/#comments</comments>
		<pubDate>Mon, 16 Nov 2009 19:59:16 +0000</pubDate>
		<dc:creator>Morgan Tocker</dc:creator>
				<category><![CDATA[mysql]]></category>
		<category><![CDATA[tips]]></category>
		<category><![CDATA[shard]]></category>
		<category><![CDATA[sharding]]></category>

		<guid isPermaLink="false">http://www.mysqlperformanceblog.com/?p=1730</guid>
		<description><![CDATA[I wrote a post a while back that said why you don&#8217;t want to shard.  In that post that I tried to explain that hardware advances such as 128G of RAM being so cheap is changing the point at which you need to shard, and that the (often omitted) operational issues created by sharding can [...]]]></description>
			<content:encoded><![CDATA[<p>I wrote a post a while back that said <a href="http://www.mysqlperformanceblog.com/2009/08/06/why-you-dont-want-to-shard/">why you don&#8217;t want to shard</a>.  In that post that I tried to explain that hardware advances such as 128G of RAM being so cheap is changing the point at which you need to shard, and that the (often omitted) operational issues created by sharding can be painful.</p>
<p>What I didn&#8217;t mention was that if you&#8217;ve established that you will need to eventually shard, is it better to just <em>get it out of the way early</em>?  My answer is <strong>almost always no</strong>. That is to say I <em>disagree</em> with a statement I&#8217;ve been hearing recently; &#8220;shard early, shard often&#8221;.  Here&#8217;s why:</p>
<ul>
<li>There&#8217;s an order of magnitude better performance that can be gained by focusing on query/index/schema optimization.  The gains from sharding are usually much lower.</li>
<li>If you shard first, and then decide you want to tune query/index/schema to reduce server count, you find yourself in a more difficult position &#8211; since you have to apply your changes across all servers.</li>
</ul>
<p><em>Or to phrase that another way:<br />
</em>I would never recommend sharding to a customer until I had <strong>at least</strong> reviewed their slow query log with <a href="http://www.maatkit.org/doc/mk-query-digest.html">mk-query-digest</a> and understood <strong>exactly</strong> why each of the queries in that report were slow.  While we have some customers who have managed to create their own tools for shard automation, it&#8217;s always easier to propose major changes to how data is stored before you have a cluster of 50+ servers.</p>
    <hr noshade style="margin:0;height:1px" />
    <p>Entry posted by Morgan Tocker |
      <a href="http://www.mysqlperformanceblog.com/2009/11/16/shard-early-shard-often/#comments">5 comments</a></p>
    <p>Add to: <a href="http://del.icio.us/post?url=http://www.mysqlperformanceblog.com/2009/11/16/shard-early-shard-often/&amp;title=&#8220;Shard early, shard often&#8221;" title="Bookmark this post on del.icio.us"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/delicious.png" alt="delicious" /></a> | <a href="http://digg.com/submit?phase=2&amp;url=http://www.mysqlperformanceblog.com/2009/11/16/shard-early-shard-often/&amp;title=&#8220;Shard early, shard often&#8221;" title="Digg this post on Digg.com"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/digg.png" alt="digg" /></a> | <a href="http://reddit.com/submit?url=http://www.mysqlperformanceblog.com/2009/11/16/shard-early-shard-often/&amp;title=&#8220;Shard early, shard often&#8221;" title="Submit this post on reddit.com"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/reddit.png" alt="reddit" /></a> | <a href="http://www.netscape.com/submit/?U=http://www.mysqlperformanceblog.com/2009/11/16/shard-early-shard-often/&amp;T=&#8220;Shard early, shard often&#8221;" title="Vote for this article on Netscape"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/netscape.gif" alt="netscape" /></a> | <a href="http://www.google.com/bookmarks/mark?op=add&amp;bkmk=http://www.mysqlperformanceblog.com/2009/11/16/shard-early-shard-often/&amp;title=&#8220;Shard early, shard often&#8221;" title="Add to Google Bookmarks"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/google.png" alt="Google Bookmarks" /></a></p>]]></content:encoded>
			<wfw:commentRss>http://www.mysqlperformanceblog.com/2009/11/16/shard-early-shard-often/feed/</wfw:commentRss>
		<slash:comments>5</slash:comments>
		</item>
		<item>
		<title>Tokyo Tyrant -The Extras Part III : Write Bottleneck</title>
		<link>http://www.mysqlperformanceblog.com/2009/11/12/tokyo-tyrant-%e2%80%93-the-extras-part-iii-write-bottleneck/</link>
		<comments>http://www.mysqlperformanceblog.com/2009/11/12/tokyo-tyrant-%e2%80%93-the-extras-part-iii-write-bottleneck/#comments</comments>
		<pubDate>Thu, 12 Nov 2009 14:00:39 +0000</pubDate>
		<dc:creator>matt</dc:creator>
				<category><![CDATA[NOSQL]]></category>
		<category><![CDATA[benchmarks]]></category>
		<category><![CDATA[mysql]]></category>
		<category><![CDATA[tips]]></category>
		<category><![CDATA[tools]]></category>

		<guid isPermaLink="false">http://www.mysqlperformanceblog.com/?p=1669</guid>
		<description><![CDATA[This is part 3 of my Tyrant extra&#8217;s, part 1 focused on durability, part 2 focused on the perceived performance wall.
#3.  Tokyo Cabinet Can have only a single writer thread, bottlenecking performance
When writing an application using Tokyo Cabinet only one connection can be opened as a “writer”  while the rest are readers.  Tyrant allows for [...]]]></description>
			<content:encoded><![CDATA[<p>This is part 3 of my Tyrant extra&#8217;s, part 1 focused on durability, part 2 focused on the perceived performance wall.</p>
<p>#3.  Tokyo Cabinet Can have only a single writer thread, bottlenecking performance</p>
<p>When writing an application using Tokyo Cabinet only one connection can be opened as a “writer”  while the rest are readers.  Tyrant allows for multiple “writes”  to be sent in from multiple applications but it still single threads them when writing out to disk.   If you run several threads all just inserting into Tyrant your will see tyrant hit 100% Cpu on 1 core, and your writes will start to peter out quickly.</p>
<p><img class="aligncenter size-full wp-image-1670" title="Single Threaded Writes" src="http://www.mysqlperformanceblog.com/wp-content/uploads/2009/11/TC_PART_2_html_m5c0e618c.gif" alt="Single Threaded Writes" width="676" height="276" /></p>
<p>In my tests when I was not disk bound (FS Cache writes) I was able to complete 4Million inserts in a little over 91 seconds using 8 threads.  I actually averaged 43896.98 inserts per second during my 8 thread test.  Moving to 10 threads doing the same 4Million inserts I completed the test in 96 seconds and averaged 41649.42 inserts per second.    Compare this to 4 Million rows using 4 threads which averaged  40933.86 and you start to see that around 40K inserts per second is the most this particular server is capable of ( single threaded ).  Hopefully this is something that maybe able to be fixed internally in the near future.  Until then you may consider breaking up your data into multiple tables each with there own cache.  This limit is per TC DB so this should work.  I had an idea about using the memcached client to distribute the data accross multiple TC database files in the back end.  This should work, I just need to test it <img src='http://www.mysqlperformanceblog.com/wp-includes/images/smilies/icon_smile.gif' alt=':)' class='wp-smiley' /> </p>
<p>Ever notice how as my multi-part posts go on they get shorter and shorter:)  This will be the last Tyrant related post for a little bit.  The 4th &amp; 5th posts were supposed to deal with replication and scaling&#8230; this may take a little while.  Thanks for reading!</p>
    <hr noshade style="margin:0;height:1px" />
    <p>Entry posted by matt |
      <a href="http://www.mysqlperformanceblog.com/2009/11/12/tokyo-tyrant-%e2%80%93-the-extras-part-iii-write-bottleneck/#comments">3 comments</a></p>
    <p>Add to: <a href="http://del.icio.us/post?url=http://www.mysqlperformanceblog.com/2009/11/12/tokyo-tyrant-%e2%80%93-the-extras-part-iii-write-bottleneck/&amp;title=Tokyo Tyrant -The Extras Part III : Write Bottleneck" title="Bookmark this post on del.icio.us"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/delicious.png" alt="delicious" /></a> | <a href="http://digg.com/submit?phase=2&amp;url=http://www.mysqlperformanceblog.com/2009/11/12/tokyo-tyrant-%e2%80%93-the-extras-part-iii-write-bottleneck/&amp;title=Tokyo Tyrant -The Extras Part III : Write Bottleneck" title="Digg this post on Digg.com"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/digg.png" alt="digg" /></a> | <a href="http://reddit.com/submit?url=http://www.mysqlperformanceblog.com/2009/11/12/tokyo-tyrant-%e2%80%93-the-extras-part-iii-write-bottleneck/&amp;title=Tokyo Tyrant -The Extras Part III : Write Bottleneck" title="Submit this post on reddit.com"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/reddit.png" alt="reddit" /></a> | <a href="http://www.netscape.com/submit/?U=http://www.mysqlperformanceblog.com/2009/11/12/tokyo-tyrant-%e2%80%93-the-extras-part-iii-write-bottleneck/&amp;T=Tokyo Tyrant -The Extras Part III : Write Bottleneck" title="Vote for this article on Netscape"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/netscape.gif" alt="netscape" /></a> | <a href="http://www.google.com/bookmarks/mark?op=add&amp;bkmk=http://www.mysqlperformanceblog.com/2009/11/12/tokyo-tyrant-%e2%80%93-the-extras-part-iii-write-bottleneck/&amp;title=Tokyo Tyrant -The Extras Part III : Write Bottleneck" title="Add to Google Bookmarks"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/google.png" alt="Google Bookmarks" /></a></p>]]></content:encoded>
			<wfw:commentRss>http://www.mysqlperformanceblog.com/2009/11/12/tokyo-tyrant-%e2%80%93-the-extras-part-iii-write-bottleneck/feed/</wfw:commentRss>
		<slash:comments>3</slash:comments>
		</item>
		<item>
		<title>Tokyo Tyrant &#8211; The Extras Part II :  The Performance Wall</title>
		<link>http://www.mysqlperformanceblog.com/2009/11/11/tokyo-tyrant-the-extras-part-ii-the-performance-wall/</link>
		<comments>http://www.mysqlperformanceblog.com/2009/11/11/tokyo-tyrant-the-extras-part-ii-the-performance-wall/#comments</comments>
		<pubDate>Wed, 11 Nov 2009 15:00:41 +0000</pubDate>
		<dc:creator>matt</dc:creator>
				<category><![CDATA[NOSQL]]></category>
		<category><![CDATA[benchmarks]]></category>
		<category><![CDATA[mysql]]></category>
		<category><![CDATA[tips]]></category>
		<category><![CDATA[tools]]></category>

		<guid isPermaLink="false">http://www.mysqlperformanceblog.com/?p=1662</guid>
		<description><![CDATA[Continuing my look at Tokyo Tyrant/Cabinet and addressing some of the concerns I have seen people have brought up this is post #2.
#2.  As your data grows does  Tokyo Cabinet slow down?
Yes your performance can degrade. One obvious performance decrease with a larger dataset  is you start to increase the likelihood that your data no [...]]]></description>
			<content:encoded><![CDATA[<p>Continuing my look at Tokyo Tyrant/Cabinet and addressing some of the concerns I have seen people have brought up this is post #2.</p>
<p>#2.  As your data grows does  Tokyo Cabinet slow down?</p>
<p>Yes your performance can degrade. One obvious performance decrease with a larger dataset  is you start to increase the likelihood that your data no longer fits into memory.  This decreases the number of memory operations and trades them for more expensive disk based operations.    As fast as any application is, as you read off disk opposed to memory performance is going to drop off substantially.  One of the more difficult things to test with Tyrant is disk bound performance.  The FS Cache can make Tyrant seem like small amounts of memory will still make it scream.  Once your data set is larger then that, people start to claim they hit the performance “wall”.</p>
<p>In order to help test this I went ahead an mounted the FS with my data files with the sync option which effectively disables the FS cache.  This should help show the real performance of the hash engine.  Here performance dips substantially, as expected :</p>
<p><img class="aligncenter size-full wp-image-1663" title="FS Mounted As Sync" src="http://www.mysqlperformanceblog.com/wp-content/uploads/2009/11/TC_PART_2_html_144b51c.gif" alt="FS Mounted As Sync" width="610" height="332" /></p>
<p>Look at the IO rate:<br />
NoSync:  31 MB/s<br />
Sync:  3.2 MB/s</p>
<p>As one would expect the IO goes crazy when the drive is mounted with the sync option hitting 99% IO wait.  The interesting this here is we are actually bottlenecking on writes and not reads.  You see without the FS cache to buffer the writes when we need to remove data from memory we now have to rely on the internal Tyrant cache and when that is exhausted have to then really write to disk not the FS Cache.  Now Tyrant starts to take on the same characteristics as your classic DB, the bigger the buffer pool the faster the performance:</p>
<p><img class="aligncenter size-full wp-image-1664" title="Difference Memory Sizes for Tyrant" src="http://www.mysqlperformanceblog.com/wp-content/uploads/2009/11/TC_PART_2_html_1bdc78f4.gif" alt="Difference Memory Sizes for Tyrant" width="585" height="372" /><br />
Even here the performance drop-off once you exhaust memory is relative.  The focus here should be the drop off versus other solutions with the same configuration, not the drop off versus a completely cached version.  In this case ask yourself given similar datasets and similar memory requirements what is the performance?  Take the above sync test, when I use 256M of memory and run my test with writes going directly to disk I hit 964 TPS, in previous MySQL tests the same setup (256M BP) netted ~160 TPS.  So 5x improvement all things being equal.  Of course this is a far drop off from the 13K I was getting when everything was effectively in the file system cache or in memory, but 5x is still a very solid improvement.</p>
<p>Next up is looking at Tyrant&#8217;s and Cabinet&#8217;s write bottleneck.</p>
    <hr noshade style="margin:0;height:1px" />
    <p>Entry posted by matt |
      <a href="http://www.mysqlperformanceblog.com/2009/11/11/tokyo-tyrant-the-extras-part-ii-the-performance-wall/#comments">3 comments</a></p>
    <p>Add to: <a href="http://del.icio.us/post?url=http://www.mysqlperformanceblog.com/2009/11/11/tokyo-tyrant-the-extras-part-ii-the-performance-wall/&amp;title=Tokyo Tyrant &#8211; The Extras Part II :  The Performance Wall" title="Bookmark this post on del.icio.us"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/delicious.png" alt="delicious" /></a> | <a href="http://digg.com/submit?phase=2&amp;url=http://www.mysqlperformanceblog.com/2009/11/11/tokyo-tyrant-the-extras-part-ii-the-performance-wall/&amp;title=Tokyo Tyrant &#8211; The Extras Part II :  The Performance Wall" title="Digg this post on Digg.com"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/digg.png" alt="digg" /></a> | <a href="http://reddit.com/submit?url=http://www.mysqlperformanceblog.com/2009/11/11/tokyo-tyrant-the-extras-part-ii-the-performance-wall/&amp;title=Tokyo Tyrant &#8211; The Extras Part II :  The Performance Wall" title="Submit this post on reddit.com"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/reddit.png" alt="reddit" /></a> | <a href="http://www.netscape.com/submit/?U=http://www.mysqlperformanceblog.com/2009/11/11/tokyo-tyrant-the-extras-part-ii-the-performance-wall/&amp;T=Tokyo Tyrant &#8211; The Extras Part II :  The Performance Wall" title="Vote for this article on Netscape"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/netscape.gif" alt="netscape" /></a> | <a href="http://www.google.com/bookmarks/mark?op=add&amp;bkmk=http://www.mysqlperformanceblog.com/2009/11/11/tokyo-tyrant-the-extras-part-ii-the-performance-wall/&amp;title=Tokyo Tyrant &#8211; The Extras Part II :  The Performance Wall" title="Add to Google Bookmarks"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/google.png" alt="Google Bookmarks" /></a></p>]]></content:encoded>
			<wfw:commentRss>http://www.mysqlperformanceblog.com/2009/11/11/tokyo-tyrant-the-extras-part-ii-the-performance-wall/feed/</wfw:commentRss>
		<slash:comments>3</slash:comments>
		</item>
		<item>
		<title>MySQL-Memcached or NOSQL Tokyo Tyrant &#8211; part 3</title>
		<link>http://www.mysqlperformanceblog.com/2009/10/19/mysql_memcached_tyrant_part3/</link>
		<comments>http://www.mysqlperformanceblog.com/2009/10/19/mysql_memcached_tyrant_part3/#comments</comments>
		<pubDate>Mon, 19 Oct 2009 16:00:43 +0000</pubDate>
		<dc:creator>matt</dc:creator>
				<category><![CDATA[NOSQL]]></category>
		<category><![CDATA[benchmarks]]></category>
		<category><![CDATA[mysql]]></category>
		<category><![CDATA[tips]]></category>
		<category><![CDATA[tuning]]></category>

		<guid isPermaLink="false">http://www.mysqlperformanceblog.com/?p=1435</guid>
		<description><![CDATA[This is part 3 of our series.  In part 1 we talked about boosting performance with memcached on top of MySQL, in Part 2 we talked about running 100% outside the data with memcached, and now in Part 3 we are going to look at a possible solution to free you from the database.  The [...]]]></description>
			<content:encoded><![CDATA[<p>This is part 3 of our series.  In <a href="http://www.mysqlperformanceblog.com/2009/10/15/mysql-memcached-or-nosql-tokyo-tyrant-part-1/">part 1</a> we talked about boosting performance with memcached on top of MySQL, in<a href="http://www.mysqlperformanceblog.com/2009/10/16/mysql_memcached_tyrant_part2/"> Part 2</a> we talked about running 100% outside the data with memcached, and now in Part 3 we are going to look at a possible solution to free you from the database.  The solution I am going to discuss here is Tokyo Cabinet and Tyrant.</p>
<p>I am not going to give you a primer  or Tutorial on Tyrant and Cabinet, there are plenty of these out there already.  Instead I want to see what sort of performance we can see compared to MySQL and Memcached, and later on other NoSQL solutions.  Tokyo actually allows you to use several types of databases that are supported, there are hash databases which are very similar to memcached, a table database which is similar to your classic database tables where you can add a where clause and search individual columns, and a ton more &#8220;database options&#8221;  beyond just those two.  Again my goal is to not make this a Tokyo Tyrant tutorial but rather show one potential role it can play.</p>
<p>More details can be read about here:<br />
<a href="http://1978th.net/tokyotyrant/"> http://1978th.net/tokyotyrant/</a><br />
<a href="http://1978th.net/tokyocabinet/"> http://1978th.net/tokyocabinet/</a></p>
<p>So if we can get performance similar to memcached with Tokoyo Tyrant when using disk based hash tables it would be a compelling replacement for our application here.  It should provide the interface and the same access we saw in memcached but with disk persistence. So let&#8217;s look at the numbers:</p>
<p><img class="aligncenter size-full wp-image-1519" title="Tyrant -vs- memcached" src="http://www.mysqlperformanceblog.com/wp-content/uploads/2009/10/more_tc_numbers_html_m31feebd3.jpg" alt="Tyrant -vs- memcached" width="662" height="354" /></p>
<p>Tyrant&#8217;s disk based hash was almost 2x faster then combining memcached and MySQL, and about 20% slower then the all memory memcached approach.  So for this particular application I would have been much better off not storing my data in MySQL and instead looking outside the database for an answer.  Now sure there are other reasons you may want to keep data in the database&#8230; but I am trying to get you to think about your application and if those reasons are really valid.  Helping clients pick the right solution is one of the things we do here at Percona.  If an application requires a database great, but if there is a better solution we want to suggest it.  It&#8217;s our goal to make your application perform optimally.</p>
<p>Finally, one concern you have to have is the scalability of your storage solution.  As load, number of threads, and data size increases how does performance differ or change?  One knock on Tokyo -vs- Memcached is Tokyo is not distributed by default.  Now that&#8217;s not to say we could not shard it based on a hash, or even build an api with the capability built in ( or use the memcached clients which works! )&#8230;  but native support is lacking.  It does support replication which could make some rather interesting architectures in the future.</p>
<p>So lets look at some scalability benchmarks, my server resources are rather limited but I thought I should try throwing more threads and work at the server until it hit its limit and fell over dead.  It&#8217;s interesting to see the number of transactions that occur with a given number of threads.  let&#8217;s look at some of these:</p>
<p><img class="size-full wp-image-1426" title="Threads &amp; Application performance" src="http://www.mysqlperformanceblog.com/wp-content/uploads/2009/10/db_right_option_html_m52d249fc.gif" alt="Tyrant/MySQL/Memcached Thread Benchmark performance" width="663" height="291" /></p>
<p>As expected the smaller buffer pool struggled ( why a smaller buffer pool?  This simulates a much larger data set.  A BP of 256M with 1GB of data, can give similar performance to 20GB of data and a 5GB BP ).  So with 256M BP and 4GB of memcached we were well off the numbers we hit with a 4GB BP+4Gb of memcached ( which is expected ).   Adding more threads even up to 128 threads increased overall throughput but my load average on the server hit 40 and my CPU was pegged.  At 128 threads I was pegging out my CPU across the board.  Also interesting is I started to hit bottlenecks in MySQL/Innodb when I had enough memory but I increased the threads from 64 to 128.  As time permits I should revisit this and look at increased datasets, and look for area&#8217;s where Tyrant may stumble a bit.</p>
<p>Bottom line given a specific application and data pattern sometimes a relational database is not the appropriate place for storing data.  A tool like Tokyo Tyrant may not be for everyone or every application, but neither is a relational database.  Before building your next application try and understand whether an RDBMS is really needed or not.</p>
<p>How did I do these tests:</p>
<p>The above number were run with 32 Threads, Tyrant was started with 8 threads and 128M of memory,  memached was started with 16 threads ( 1.4 memcached ), mysql was 5.1 XtraDB.  Each environment had 2 tables each with 2 million rows.  The data was identical. memcached and Tyrant stored a comma delimited string to represent the row.   Mysql was running with 256M allocated to the innodb buffer unless otherwise noted.</p>
<p>What&#8217;s next?  Well next I am going to try and continue this series by exploring and benchmarking other NOSQL options and comparing them to database based solutions.  I think showing the performance of a couple of different Tokyo database formats would also be interesting.  What other solutions are people interested in?  I know I have gotten a lot of requests for cassandra #&#8217;s, but what else?  Drop a comment and let me know!</p>
    <hr noshade style="margin:0;height:1px" />
    <p>Entry posted by matt |
      <a href="http://www.mysqlperformanceblog.com/2009/10/19/mysql_memcached_tyrant_part3/#comments">24 comments</a></p>
    <p>Add to: <a href="http://del.icio.us/post?url=http://www.mysqlperformanceblog.com/2009/10/19/mysql_memcached_tyrant_part3/&amp;title=MySQL-Memcached or NOSQL Tokyo Tyrant &#8211; part 3" title="Bookmark this post on del.icio.us"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/delicious.png" alt="delicious" /></a> | <a href="http://digg.com/submit?phase=2&amp;url=http://www.mysqlperformanceblog.com/2009/10/19/mysql_memcached_tyrant_part3/&amp;title=MySQL-Memcached or NOSQL Tokyo Tyrant &#8211; part 3" title="Digg this post on Digg.com"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/digg.png" alt="digg" /></a> | <a href="http://reddit.com/submit?url=http://www.mysqlperformanceblog.com/2009/10/19/mysql_memcached_tyrant_part3/&amp;title=MySQL-Memcached or NOSQL Tokyo Tyrant &#8211; part 3" title="Submit this post on reddit.com"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/reddit.png" alt="reddit" /></a> | <a href="http://www.netscape.com/submit/?U=http://www.mysqlperformanceblog.com/2009/10/19/mysql_memcached_tyrant_part3/&amp;T=MySQL-Memcached or NOSQL Tokyo Tyrant &#8211; part 3" title="Vote for this article on Netscape"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/netscape.gif" alt="netscape" /></a> | <a href="http://www.google.com/bookmarks/mark?op=add&amp;bkmk=http://www.mysqlperformanceblog.com/2009/10/19/mysql_memcached_tyrant_part3/&amp;title=MySQL-Memcached or NOSQL Tokyo Tyrant &#8211; part 3" title="Add to Google Bookmarks"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/google.png" alt="Google Bookmarks" /></a></p>]]></content:encoded>
			<wfw:commentRss>http://www.mysqlperformanceblog.com/2009/10/19/mysql_memcached_tyrant_part3/feed/</wfw:commentRss>
		<slash:comments>24</slash:comments>
		</item>
		<item>
		<title>How (not) to find unused indexes</title>
		<link>http://www.mysqlperformanceblog.com/2009/10/16/how-not-to-find-unused-indexes/</link>
		<comments>http://www.mysqlperformanceblog.com/2009/10/16/how-not-to-find-unused-indexes/#comments</comments>
		<pubDate>Fri, 16 Oct 2009 16:02:38 +0000</pubDate>
		<dc:creator>Morgan Tocker</dc:creator>
				<category><![CDATA[mysql]]></category>
		<category><![CDATA[tips]]></category>
		<category><![CDATA[explain]]></category>
		<category><![CDATA[indexing]]></category>

		<guid isPermaLink="false">http://www.mysqlperformanceblog.com/?p=1243</guid>
		<description><![CDATA[I've seen a few people link to an INFORMATION_SCHEMA query to be able to find any indexes that have low cardinality, in an effort to find out what indexes should be removed.  This method is flawed - here's the first reason why:
PLAIN TEXT
SQL:




CREATE TABLE `sales` &#40;


`id` int&#40;11&#41; NOT NULL AUTO_INCREMENT,


`customer_id` int&#40;11&#41; DEFAULT NULL,


`status` enum&#40;'archived','active'&#41; DEFAULT [...]]]></description>
			<content:encoded><![CDATA[<p>I've seen a few people link to an INFORMATION_SCHEMA query to be able to find any indexes that have low cardinality, in an effort to find out what indexes should be removed.  This method is <em>flawed</em> - here's the <strong>first reason</strong> why:</p>
<div class="igBar"><span id="lsql-3"><a href="#" onclick="javascript:showPlainTxt('sql-3'); return false;">PLAIN TEXT</a></span></div>
<div class="syntax_hilite"><span class="langName">SQL:</span>
<div id="sql-3">
<div class="sql">
<ol>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color: #993333; font-weight: bold;">CREATE</span> <span style="color: #993333; font-weight: bold;">TABLE</span> <span style="color: #ff0000;">`sales`</span> <span style="color:#006600; font-weight:bold;">&#40;</span></div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color: #ff0000;">`id`</span> int<span style="color:#006600; font-weight:bold;">&#40;</span><span style="color: #cc66cc;color:#800000;">11</span><span style="color:#006600; font-weight:bold;">&#41;</span> <span style="color: #993333; font-weight: bold;">NOT</span> <span style="color: #993333; font-weight: bold;">NULL</span> <span style="color: #993333; font-weight: bold;">AUTO_INCREMENT</span>,</div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color: #ff0000;">`customer_id`</span> int<span style="color:#006600; font-weight:bold;">&#40;</span><span style="color: #cc66cc;color:#800000;">11</span><span style="color:#006600; font-weight:bold;">&#41;</span> <span style="color: #993333; font-weight: bold;">DEFAULT</span> <span style="color: #993333; font-weight: bold;">NULL</span>,</div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color: #ff0000;">`status`</span> enum<span style="color:#006600; font-weight:bold;">&#40;</span><span style="color: #ff0000;">'archived'</span>,<span style="color: #ff0000;">'active'</span><span style="color:#006600; font-weight:bold;">&#41;</span> <span style="color: #993333; font-weight: bold;">DEFAULT</span> <span style="color: #993333; font-weight: bold;">NULL</span>,</div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color: #993333; font-weight: bold;">PRIMARY</span> <span style="color: #993333; font-weight: bold;">KEY</span> <span style="color:#006600; font-weight:bold;">&#40;</span><span style="color: #ff0000;">`id`</span><span style="color:#006600; font-weight:bold;">&#41;</span>,</div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color: #993333; font-weight: bold;">KEY</span> <span style="color: #ff0000;">`status`</span> <span style="color:#006600; font-weight:bold;">&#40;</span><span style="color: #ff0000;">`status`</span><span style="color:#006600; font-weight:bold;">&#41;</span></div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color:#006600; font-weight:bold;">&#41;</span> ENGINE=MyISAM <span style="color: #993333; font-weight: bold;">AUTO_INCREMENT</span>=<span style="color: #cc66cc;color:#800000;">65691</span> <span style="color: #993333; font-weight: bold;">DEFAULT</span> CHARSET=latin1;</div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp;</div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">mysql&amp;gt; <span style="color: #993333; font-weight: bold;">SELECT</span> count<span style="color:#006600; font-weight:bold;">&#40;</span>*<span style="color:#006600; font-weight:bold;">&#41;</span>, <span style="color: #993333; font-weight: bold;">STATUS</span> <span style="color: #993333; font-weight: bold;">FROM</span> sales <span style="color: #993333; font-weight: bold;">GROUP</span> <span style="color: #993333; font-weight: bold;">BY</span> <span style="color: #993333; font-weight: bold;">STATUS</span>;</div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">+<span style="color: #808080; font-style: italic;">----------+---------+</span></div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">| count<span style="color:#006600; font-weight:bold;">&#40;</span>*<span style="color:#006600; font-weight:bold;">&#41;</span> | <span style="color: #993333; font-weight: bold;">STATUS</span>  |</div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">+<span style="color: #808080; font-style: italic;">----------+---------+</span></div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">|    <span style="color: #cc66cc;color:#800000;">65536</span> | archived |</div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">|      <span style="color: #cc66cc;color:#800000;">154</span> | active  |</div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">+<span style="color: #808080; font-style: italic;">----------+---------+</span></div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color: #cc66cc;color:#800000;">2</span> rows <span style="color: #993333; font-weight: bold;">IN</span> <span style="color: #993333; font-weight: bold;">SET</span> <span style="color:#006600; font-weight:bold;">&#40;</span><span style="color: #cc66cc;color:#800000;">0</span>.<span style="color: #cc66cc;color:#800000;">17</span> sec<span style="color:#006600; font-weight:bold;">&#41;</span></div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp;</div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">mysql&amp;gt; <span style="color: #993333; font-weight: bold;">EXPLAIN</span> <span style="color: #993333; font-weight: bold;">SELECT</span> * <span style="color: #993333; font-weight: bold;">FROM</span> sales <span style="color: #993333; font-weight: bold;">WHERE</span> <span style="color: #993333; font-weight: bold;">STATUS</span>=<span style="color: #ff0000;">'active'</span>; <span style="color: #808080; font-style: italic;"># query 1</span></div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">+<span style="color: #808080; font-style: italic;">----+-------------+-------+------+---------------+--------+---------+-------+------+-------------+</span></div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">| id | select_type | <span style="color: #993333; font-weight: bold;">TABLE</span> | type | possible_keys | <span style="color: #993333; font-weight: bold;">KEY</span>    | key_len | ref   | rows | Extra       |</div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">+<span style="color: #808080; font-style: italic;">----+-------------+-------+------+---------------+--------+---------+-------+------+-------------+</span></div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">|  <span style="color: #cc66cc;color:#800000;">1</span> | SIMPLE      | sales | ref  | <span style="color: #993333; font-weight: bold;">STATUS</span>        | <span style="color: #993333; font-weight: bold;">STATUS</span> | <span style="color: #cc66cc;color:#800000;">2</span>       | const |  <span style="color: #cc66cc;color:#800000;">196</span> | <span style="color: #993333; font-weight: bold;">USING</span> <span style="color: #993333; font-weight: bold;">WHERE</span> |</div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">+<span style="color: #808080; font-style: italic;">----+-------------+-------+------+---------------+--------+---------+-------+------+-------------+</span></div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color: #cc66cc;color:#800000;">1</span> row <span style="color: #993333; font-weight: bold;">IN</span> <span style="color: #993333; font-weight: bold;">SET</span> <span style="color:#006600; font-weight:bold;">&#40;</span><span style="color: #cc66cc;color:#800000;">0</span>.<span style="color: #cc66cc;color:#800000;">06</span> sec<span style="color:#006600; font-weight:bold;">&#41;</span></div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp;</div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">mysql&amp;gt; <span style="color: #993333; font-weight: bold;">EXPLAIN</span> <span style="color: #993333; font-weight: bold;">SELECT</span> * <span style="color: #993333; font-weight: bold;">FROM</span> sales <span style="color: #993333; font-weight: bold;">WHERE</span> <span style="color: #993333; font-weight: bold;">STATUS</span>=<span style="color: #ff0000;">'archived'</span>; <span style="color: #808080; font-style: italic;"># query 2</span></div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">+<span style="color: #808080; font-style: italic;">----+-------------+-------+------+---------------+------+---------+------+-------+-------------+</span></div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">| id | select_type | <span style="color: #993333; font-weight: bold;">TABLE</span> | type | possible_keys | <span style="color: #993333; font-weight: bold;">KEY</span>  | key_len | ref  | rows  | Extra       |</div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">+<span style="color: #808080; font-style: italic;">----+-------------+-------+------+---------------+------+---------+------+-------+-------------+</span></div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">|  <span style="color: #cc66cc;color:#800000;">1</span> | SIMPLE      | sales | <span style="color: #993333; font-weight: bold;">ALL</span>  | <span style="color: #993333; font-weight: bold;">STATUS</span>        | <span style="color: #993333; font-weight: bold;">NULL</span> | <span style="color: #993333; font-weight: bold;">NULL</span>    | <span style="color: #993333; font-weight: bold;">NULL</span> | <span style="color: #cc66cc;color:#800000;">65690</span> | <span style="color: #993333; font-weight: bold;">USING</span> <span style="color: #993333; font-weight: bold;">WHERE</span> |</div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">+<span style="color: #808080; font-style: italic;">----+-------------+-------+------+---------------+------+---------+------+-------+-------------+</span></div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color: #cc66cc;color:#800000;">1</span> row <span style="color: #993333; font-weight: bold;">IN</span> <span style="color: #993333; font-weight: bold;">SET</span> <span style="color:#006600; font-weight:bold;">&#40;</span><span style="color: #cc66cc;color:#800000;">0</span>.<span style="color: #cc66cc;color:#800000;">01</span> sec<span style="color:#006600; font-weight:bold;">&#41;</span> </div>
</li>
</ol>
</div>
</div>
</div>
<p></p>
<p>The cardinality of status index is woeful, but provided that the application is always only sending query 1 to MySQL it's actually a pretty good index!  It's not always like this, but there are a lot of cases where applications have good selectivity with some queries despite what cardinality shows.</p>
<p>Not convinced?  Here's <strong>reason number two</strong>:</p>
<div class="igBar"><span id="lsql-4"><a href="#" onclick="javascript:showPlainTxt('sql-4'); return false;">PLAIN TEXT</a></span></div>
<div class="syntax_hilite"><span class="langName">SQL:</span>
<div id="sql-4">
<div class="sql">
<ol>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color: #993333; font-weight: bold;">CREATE</span> <span style="color: #993333; font-weight: bold;">TABLE</span> <span style="color: #ff0000;">`Country`</span> <span style="color:#006600; font-weight:bold;">&#40;</span></div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color: #ff0000;">`Code`</span> char<span style="color:#006600; font-weight:bold;">&#40;</span><span style="color: #cc66cc;color:#800000;">3</span><span style="color:#006600; font-weight:bold;">&#41;</span> <span style="color: #993333; font-weight: bold;">NOT</span> <span style="color: #993333; font-weight: bold;">NULL</span> <span style="color: #993333; font-weight: bold;">DEFAULT</span> <span style="color: #ff0000;">''</span>,</div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color: #ff0000;">`Name`</span> char<span style="color:#006600; font-weight:bold;">&#40;</span><span style="color: #cc66cc;color:#800000;">52</span><span style="color:#006600; font-weight:bold;">&#41;</span> <span style="color: #993333; font-weight: bold;">NOT</span> <span style="color: #993333; font-weight: bold;">NULL</span> <span style="color: #993333; font-weight: bold;">DEFAULT</span> <span style="color: #ff0000;">''</span>,</div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color: #ff0000;">`Continent`</span> enum<span style="color:#006600; font-weight:bold;">&#40;</span><span style="color: #ff0000;">'Asia'</span>,<span style="color: #ff0000;">'Europe'</span>,<span style="color: #ff0000;">'North America'</span>,<span style="color: #ff0000;">'Africa'</span>,<span style="color: #ff0000;">'Oceania'</span>,<span style="color: #ff0000;">'Antarctica'</span>,<span style="color: #ff0000;">'South America'</span><span style="color:#006600; font-weight:bold;">&#41;</span> <span style="color: #993333; font-weight: bold;">NOT</span> <span style="color: #993333; font-weight: bold;">NULL</span> <span style="color: #993333; font-weight: bold;">DEFAULT</span> <span style="color: #ff0000;">'Asia'</span>,</div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color: #ff0000;">`Region`</span> char<span style="color:#006600; font-weight:bold;">&#40;</span><span style="color: #cc66cc;color:#800000;">26</span><span style="color:#006600; font-weight:bold;">&#41;</span> <span style="color: #993333; font-weight: bold;">NOT</span> <span style="color: #993333; font-weight: bold;">NULL</span> <span style="color: #993333; font-weight: bold;">DEFAULT</span> <span style="color: #ff0000;">''</span>,</div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color: #ff0000;">`SurfaceArea`</span> float<span style="color:#006600; font-weight:bold;">&#40;</span><span style="color: #cc66cc;color:#800000;">10</span>,<span style="color: #cc66cc;color:#800000;">2</span><span style="color:#006600; font-weight:bold;">&#41;</span> <span style="color: #993333; font-weight: bold;">NOT</span> <span style="color: #993333; font-weight: bold;">NULL</span> <span style="color: #993333; font-weight: bold;">DEFAULT</span> <span style="color: #ff0000;">'0.00'</span>,</div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color: #ff0000;">`IndepYear`</span> smallint<span style="color:#006600; font-weight:bold;">&#40;</span><span style="color: #cc66cc;color:#800000;">6</span><span style="color:#006600; font-weight:bold;">&#41;</span> <span style="color: #993333; font-weight: bold;">DEFAULT</span> <span style="color: #993333; font-weight: bold;">NULL</span>,</div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color: #ff0000;">`Population`</span> int<span style="color:#006600; font-weight:bold;">&#40;</span><span style="color: #cc66cc;color:#800000;">11</span><span style="color:#006600; font-weight:bold;">&#41;</span> <span style="color: #993333; font-weight: bold;">NOT</span> <span style="color: #993333; font-weight: bold;">NULL</span> <span style="color: #993333; font-weight: bold;">DEFAULT</span> <span style="color: #ff0000;">'0'</span>,</div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color: #ff0000;">`LifeExpectancy`</span> float<span style="color:#006600; font-weight:bold;">&#40;</span><span style="color: #cc66cc;color:#800000;">3</span>,<span style="color: #cc66cc;color:#800000;">1</span><span style="color:#006600; font-weight:bold;">&#41;</span> <span style="color: #993333; font-weight: bold;">DEFAULT</span> <span style="color: #993333; font-weight: bold;">NULL</span>,</div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color: #ff0000;">`GNP`</span> float<span style="color:#006600; font-weight:bold;">&#40;</span><span style="color: #cc66cc;color:#800000;">10</span>,<span style="color: #cc66cc;color:#800000;">2</span><span style="color:#006600; font-weight:bold;">&#41;</span> <span style="color: #993333; font-weight: bold;">DEFAULT</span> <span style="color: #993333; font-weight: bold;">NULL</span>,</div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color: #ff0000;">`GNPOld`</span> float<span style="color:#006600; font-weight:bold;">&#40;</span><span style="color: #cc66cc;color:#800000;">10</span>,<span style="color: #cc66cc;color:#800000;">2</span><span style="color:#006600; font-weight:bold;">&#41;</span> <span style="color: #993333; font-weight: bold;">DEFAULT</span> <span style="color: #993333; font-weight: bold;">NULL</span>,</div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color: #ff0000;">`LocalName`</span> char<span style="color:#006600; font-weight:bold;">&#40;</span><span style="color: #cc66cc;color:#800000;">45</span><span style="color:#006600; font-weight:bold;">&#41;</span> <span style="color: #993333; font-weight: bold;">NOT</span> <span style="color: #993333; font-weight: bold;">NULL</span> <span style="color: #993333; font-weight: bold;">DEFAULT</span> <span style="color: #ff0000;">''</span>,</div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color: #ff0000;">`GovernmentForm`</span> char<span style="color:#006600; font-weight:bold;">&#40;</span><span style="color: #cc66cc;color:#800000;">45</span><span style="color:#006600; font-weight:bold;">&#41;</span> <span style="color: #993333; font-weight: bold;">NOT</span> <span style="color: #993333; font-weight: bold;">NULL</span> <span style="color: #993333; font-weight: bold;">DEFAULT</span> <span style="color: #ff0000;">''</span>,</div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color: #ff0000;">`HeadOfState`</span> char<span style="color:#006600; font-weight:bold;">&#40;</span><span style="color: #cc66cc;color:#800000;">60</span><span style="color:#006600; font-weight:bold;">&#41;</span> <span style="color: #993333; font-weight: bold;">DEFAULT</span> <span style="color: #993333; font-weight: bold;">NULL</span>,</div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color: #ff0000;">`Capital`</span> int<span style="color:#006600; font-weight:bold;">&#40;</span><span style="color: #cc66cc;color:#800000;">11</span><span style="color:#006600; font-weight:bold;">&#41;</span> <span style="color: #993333; font-weight: bold;">DEFAULT</span> <span style="color: #993333; font-weight: bold;">NULL</span>,</div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color: #ff0000;">`Code2`</span> char<span style="color:#006600; font-weight:bold;">&#40;</span><span style="color: #cc66cc;color:#800000;">2</span><span style="color:#006600; font-weight:bold;">&#41;</span> <span style="color: #993333; font-weight: bold;">NOT</span> <span style="color: #993333; font-weight: bold;">NULL</span> <span style="color: #993333; font-weight: bold;">DEFAULT</span> <span style="color: #ff0000;">''</span>,</div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color: #993333; font-weight: bold;">PRIMARY</span> <span style="color: #993333; font-weight: bold;">KEY</span> <span style="color:#006600; font-weight:bold;">&#40;</span><span style="color: #ff0000;">`Code`</span><span style="color:#006600; font-weight:bold;">&#41;</span>,</div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color: #993333; font-weight: bold;">KEY</span> <span style="color: #ff0000;">`Population`</span> <span style="color:#006600; font-weight:bold;">&#40;</span><span style="color: #ff0000;">`Population`</span><span style="color:#006600; font-weight:bold;">&#41;</span></div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color:#006600; font-weight:bold;">&#41;</span> ENGINE=MyISAM <span style="color: #993333; font-weight: bold;">DEFAULT</span> CHARSET=latin1;</div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp;</div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">mysql&amp;gt; <span style="color: #993333; font-weight: bold;">SELECT</span> count<span style="color:#006600; font-weight:bold;">&#40;</span>*<span style="color:#006600; font-weight:bold;">&#41;</span> <span style="color: #993333; font-weight: bold;">FROM</span> Country;</div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">+<span style="color: #808080; font-style: italic;">----------+</span></div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">| count<span style="color:#006600; font-weight:bold;">&#40;</span>*<span style="color:#006600; font-weight:bold;">&#41;</span> |</div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">+<span style="color: #808080; font-style: italic;">----------+</span></div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">|      <span style="color: #cc66cc;color:#800000;">239</span> |</div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">+<span style="color: #808080; font-style: italic;">----------+</span></div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color: #cc66cc;color:#800000;">1</span> row <span style="color: #993333; font-weight: bold;">IN</span> <span style="color: #993333; font-weight: bold;">SET</span> <span style="color:#006600; font-weight:bold;">&#40;</span><span style="color: #cc66cc;color:#800000;">0</span>.<span style="color: #cc66cc;color:#800000;">00</span> sec<span style="color:#006600; font-weight:bold;">&#41;</span></div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp;</div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">mysql&amp;gt; <span style="color: #993333; font-weight: bold;">SELECT</span> count<span style="color:#006600; font-weight:bold;">&#40;</span><span style="color: #993333; font-weight: bold;">DISTINCT</span><span style="color:#006600; font-weight:bold;">&#40;</span>population<span style="color:#006600; font-weight:bold;">&#41;</span><span style="color:#006600; font-weight:bold;">&#41;</span> <span style="color: #993333; font-weight: bold;">FROM</span> Country;</div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">+<span style="color: #808080; font-style: italic;">-----------------------------+</span></div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">| count<span style="color:#006600; font-weight:bold;">&#40;</span><span style="color: #993333; font-weight: bold;">DISTINCT</span><span style="color:#006600; font-weight:bold;">&#40;</span>population<span style="color:#006600; font-weight:bold;">&#41;</span><span style="color:#006600; font-weight:bold;">&#41;</span> |</div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">+<span style="color: #808080; font-style: italic;">-----------------------------+</span></div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">|                         <span style="color: #cc66cc;color:#800000;">226</span> |</div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">+<span style="color: #808080; font-style: italic;">-----------------------------+</span></div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color: #cc66cc;color:#800000;">1</span> row <span style="color: #993333; font-weight: bold;">IN</span> <span style="color: #993333; font-weight: bold;">SET</span> <span style="color:#006600; font-weight:bold;">&#40;</span><span style="color: #cc66cc;color:#800000;">0</span>.<span style="color: #cc66cc;color:#800000;">05</span> sec<span style="color:#006600; font-weight:bold;">&#41;</span></div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp;</div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">mysql&amp;gt; <span style="color: #993333; font-weight: bold;">EXPLAIN</span> <span style="color: #993333; font-weight: bold;">SELECT</span> * <span style="color: #993333; font-weight: bold;">FROM</span> country <span style="color: #993333; font-weight: bold;">WHERE</span> population&amp;gt; <span style="color: #cc66cc;color:#800000;">1000</span>; <span style="color: #808080; font-style: italic;"># query 3</span></div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">+<span style="color: #808080; font-style: italic;">----+-------------+---------+------+---------------+------+---------+------+------+-------------+</span></div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">| id | select_type | <span style="color: #993333; font-weight: bold;">TABLE</span>   | type | possible_keys | <span style="color: #993333; font-weight: bold;">KEY</span>  | key_len | ref  | rows | Extra       |</div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">+<span style="color: #808080; font-style: italic;">----+-------------+---------+------+---------------+------+---------+------+------+-------------+</span></div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">|  <span style="color: #cc66cc;color:#800000;">1</span> | SIMPLE      | country | <span style="color: #993333; font-weight: bold;">ALL</span>  | Population    | <span style="color: #993333; font-weight: bold;">NULL</span> | <span style="color: #993333; font-weight: bold;">NULL</span>    | <span style="color: #993333; font-weight: bold;">NULL</span> |  <span style="color: #cc66cc;color:#800000;">239</span> | <span style="color: #993333; font-weight: bold;">USING</span> <span style="color: #993333; font-weight: bold;">WHERE</span> |</div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">+<span style="color: #808080; font-style: italic;">----+-------------+---------+------+---------------+------+---------+------+------+-------------+</span></div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color: #cc66cc;color:#800000;">1</span> row <span style="color: #993333; font-weight: bold;">IN</span> <span style="color: #993333; font-weight: bold;">SET</span> <span style="color:#006600; font-weight:bold;">&#40;</span><span style="color: #cc66cc;color:#800000;">0</span>.<span style="color: #cc66cc;color:#800000;">04</span> sec<span style="color:#006600; font-weight:bold;">&#41;</span></div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp;</div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">mysql&amp;gt; <span style="color: #993333; font-weight: bold;">EXPLAIN</span> <span style="color: #993333; font-weight: bold;">SELECT</span> * <span style="color: #993333; font-weight: bold;">FROM</span> country <span style="color: #993333; font-weight: bold;">WHERE</span> population&amp;gt; <span style="color: #cc66cc;color:#800000;">100000000</span>; <span style="color: #808080; font-style: italic;"># query 4</span></div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">+<span style="color: #808080; font-style: italic;">----+-------------+---------+-------+---------------+------------+---------+------+------+-------------+</span></div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">| id | select_type | <span style="color: #993333; font-weight: bold;">TABLE</span>   | type  | possible_keys | <span style="color: #993333; font-weight: bold;">KEY</span>        | key_len | ref  | rows | Extra       |</div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">+<span style="color: #808080; font-style: italic;">----+-------------+---------+-------+---------------+------------+---------+------+------+-------------+</span></div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">|  <span style="color: #cc66cc;color:#800000;">1</span> | SIMPLE      | country | range | Population    | Population | <span style="color: #cc66cc;color:#800000;">4</span>       | <span style="color: #993333; font-weight: bold;">NULL</span> |   <span style="color: #cc66cc;color:#800000;">23</span> | <span style="color: #993333; font-weight: bold;">USING</span> <span style="color: #993333; font-weight: bold;">WHERE</span> |</div>
</li>
<li style="font-weight: bold;color:#26536A;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">+<span style="color: #808080; font-style: italic;">----+-------------+---------+-------+---------------+------------+---------+------+------+-------------+</span></div>
</li>
<li style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;">
<div style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span style="color: #cc66cc;color:#800000;">1</span> row <span style="color: #993333; font-weight: bold;">IN</span> <span style="color: #993333; font-weight: bold;">SET</span> <span style="color:#006600; font-weight:bold;">&#40;</span><span style="color: #cc66cc;color:#800000;">0</span>.<span style="color: #cc66cc;color:#800000;">00</span> sec<span style="color:#006600; font-weight:bold;">&#41;</span> </div>
</li>
</ol>
</div>
</div>
</div>
<p></p>
<p>The index on query 3 had high cardinality but should not be used since too many countries have a population greater than 1000.  An automated search for low cardinality indexes wouldn't have revealed it's uselessness.  For range scans, it's very easy to lead yourself into a trap where your index can not filter out enough rows to be effective.  I see this a lot in consulting issues where customers have queries that use a BETWEEN on a date, but the window of time it is searching in is too wide.</p>
<p><strong>Side Note:</strong> In some texts you'll see people quote the numbers "20-30%" as the minimum amount of rows you have to filter down to for an index to be useful (that is, eliminate 70-80% of rows).  It's not quite correct to quote this as an exact percentage, since this value is not fixed in MySQL and can be a much wider window (15-60%) depending on the circumstances.  In this case, MySQL flipped from tablescan to index at about 34%.</p>
<p><strong>How am I supposed to find unused indexes then?<br />
</strong>You really have to run queries against your server - <strong>there is no other way</strong>.  From there, there's a helpful patch in <a href="http://www.percona.com/docs/wiki/release:start">5.0-percona</a> called <a href="http://www.percona.com/docs/wiki/patches:userstatv2">INDEX_STATISTICS</a> that can then show you which indexes were touched and which were not.</p>
<p>If you are not running a patched server, then the alternative is to either use a proxy that checks EXPLAIN information (like <a href="http://www.mysql.com/products/enterprise/query.html">QUAN</a>) or set your slow query log to zero microseconds (5.1 feature) and then find someway to parse and EXPLAIN all results, then subtract the indexes that were mentioned from all indexes known.  There's an old tool called <a href="http://hackmysql.com/mysqlidxchk">mysqlidxchx</a> which should be able to do this.</p>
    <hr noshade style="margin:0;height:1px" />
    <p>Entry posted by Morgan Tocker |
      <a href="http://www.mysqlperformanceblog.com/2009/10/16/how-not-to-find-unused-indexes/#comments">One comment</a></p>
    <p>Add to: <a href="http://del.icio.us/post?url=http://www.mysqlperformanceblog.com/2009/10/16/how-not-to-find-unused-indexes/&amp;title=How (not) to find unused indexes" title="Bookmark this post on del.icio.us"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/delicious.png" alt="delicious" /></a> | <a href="http://digg.com/submit?phase=2&amp;url=http://www.mysqlperformanceblog.com/2009/10/16/how-not-to-find-unused-indexes/&amp;title=How (not) to find unused indexes" title="Digg this post on Digg.com"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/digg.png" alt="digg" /></a> | <a href="http://reddit.com/submit?url=http://www.mysqlperformanceblog.com/2009/10/16/how-not-to-find-unused-indexes/&amp;title=How (not) to find unused indexes" title="Submit this post on reddit.com"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/reddit.png" alt="reddit" /></a> | <a href="http://www.netscape.com/submit/?U=http://www.mysqlperformanceblog.com/2009/10/16/how-not-to-find-unused-indexes/&amp;T=How (not) to find unused indexes" title="Vote for this article on Netscape"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/netscape.gif" alt="netscape" /></a> | <a href="http://www.google.com/bookmarks/mark?op=add&amp;bkmk=http://www.mysqlperformanceblog.com/2009/10/16/how-not-to-find-unused-indexes/&amp;title=How (not) to find unused indexes" title="Add to Google Bookmarks"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/google.png" alt="Google Bookmarks" /></a></p>]]></content:encoded>
			<wfw:commentRss>http://www.mysqlperformanceblog.com/2009/10/16/how-not-to-find-unused-indexes/feed/</wfw:commentRss>
		<slash:comments>1</slash:comments>
		</item>
		<item>
		<title>MySQL-Memcached or NOSQL Tokyo Tyrant &#8211; part 2</title>
		<link>http://www.mysqlperformanceblog.com/2009/10/16/mysql_memcached_tyrant_part2/</link>
		<comments>http://www.mysqlperformanceblog.com/2009/10/16/mysql_memcached_tyrant_part2/#comments</comments>
		<pubDate>Fri, 16 Oct 2009 16:00:41 +0000</pubDate>
		<dc:creator>matt</dc:creator>
				<category><![CDATA[NOSQL]]></category>
		<category><![CDATA[benchmarks]]></category>
		<category><![CDATA[memcached]]></category>
		<category><![CDATA[mysql]]></category>
		<category><![CDATA[tips]]></category>

		<guid isPermaLink="false">http://www.mysqlperformanceblog.com/?p=1433</guid>
		<description><![CDATA[Part 1 of our series set-up our "test"  application and looked at boosting performance of the application by buffer MySQL with memcached.  Our test application is simple and requires only 3 basic operations per transaction 2 reads and 1 write.  Using memcached combined with MySQL we ended up nearly getting a 10X performance boost from [...]]]></description>
			<content:encoded><![CDATA[<p><a href="http://www.mysqlperformanceblog.com/2009/10/15/mysql-memcached-or-nosql-tokyo-tyrant-part-1/">Part 1</a> of our series set-up our "test"  application and looked at boosting performance of the application by buffer MySQL with memcached.  Our test application is simple and requires only 3 basic operations per transaction 2 reads and 1 write.  Using memcached combined with MySQL we ended up nearly getting a 10X performance boost from the application.  Now we are going to look at what we could achieve if we did not have to write to the database at all.  So let's look at what happens if we push everything including writes into memcached.</p>
<p><img class="size-full wp-image-1427" title="Benchmarks if everything is in memcached" src="http://www.mysqlperformanceblog.com/wp-content/uploads/2009/10/db_right_option_html_m62d9ce6b.gif" alt="Benchmarks if everything is in memcached" width="641" height="335" /></p>
<p>Wow that's shockingly fast isn't it! I guess being completely in memory helps for this app.  What is very interesting is accessing 100% of the data in memcached gives very similar numbers to accessing 100% of the data in memory in the DB ( part 1 benchmarked a 4GB bp as being able to handle 7K TPS)... something is not 100% right here.  It stands to reason that memcached should be faster for this application then the DB.  Its just doing two gets via key and 1 set.  So why the similar numbers?</p>
<p>Well glad you asked.  It's the API.  The api in this case was Cache::Memcached, by switching to using Cache::Memcached::Fast look what happens:</p>
<p><img class="size-full wp-image-1428" title="Memcached API - Fast" src="http://www.mysqlperformanceblog.com/wp-content/uploads/2009/10/db_right_option_html_m74e8a5a8.gif" alt="Memcached API - Fast" width="704" height="317" /></p>
<p>That is a nice jump in performance!</p>
<p>Using Memcached::Fast was kind of a mixed bag when looking at the benchmarks for mixing MySQL and Memcached in my tests:</p>
<p><img class="size-full wp-image-1421" title="Memcached Api's" src="http://www.mysqlperformanceblog.com/wp-content/uploads/2009/10/db_right_option_html_2ff88cd9.gif" alt="Sometimes Api changes can make a huge difference" width="667" height="340" /></p>
<p>In this case I think the Fast api was slower when working with MySQL with a 256m BP because the slower returns from memcached acted as a bottleneck to thin the demands on MySQL to write data, smoothing out the work load.  When we eliminate this bottleneck with the Fast api, MySQL gets hammered.  This type of thing happens a lot.  For example an application is CPU bound, so you add more processing power, but then you hit disks harder and  now your disk bound.</p>
<p>A couple of good things to remember here:  #1 resolving 1 bottleneck can open another bottleneck that is much worse.  #2  is to understand that not all API's are created equal.  Additionally the configuration and setup that works well on one system may not work well on another.  Because of this people often leave lots of performance on the table.  Don't just trust that your current API or config is optimal, test and make sure it fits your application.</p>
<p>So adding Memcached on top of MySQL for our test application can significantly boost performance. But you notice that if we were running 100% in memcached and could cut out MySQL we could get 2.5x more performance over a mixed solution and 100X over just stock MySQL.  As the number of writes against the database increase this gap will increase.  So let's ditch the database!  But wait!  you need the DB for  persistence, right?</p>
<p>It depends.  A database may not be the best fit for every application.  There are several “NOSQL”  solutions out in the open source space that can give you some of the ease of a Memcached but with persistence most people use their database for.   Each application is different and understanding the application's requirements is key to picking an appropriate solution.   I am going to look at several database alternatives over the next few months.  I need to start somewhere, so I decided to start with Tokyo Tyrant and Cabinet.    So stop in next time for part 3 of this series where we will focus on running the same tests against Tokyo Tyrant.</p>
<div id="_mcePaste" style="overflow: hidden; position: absolute; left: -10000px; top: 0px; width: 1px; height: 1px;"><!-- 		@page { margin: 0.79in } 		P { margin-bottom: 0.08in } --></p>
<p style="margin-bottom: 0in;">Wow that's shockingly fast isn't it! I guess being completely in memory helps for this app.  What is very interesting is accessing 100% of the data in memcached gives very similar numbers to accessing 100% of the data in memory in the DB... something is not 100% right here.  It stands to reason that memcached should be faster for this application then the DB, two gets via key and 1 set.  So why the similar numbers?</p>
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">Well glad you asked.  It's the API.  The api in this case was Cache::Memcached, by switching to using Cache::Memcached::Fast look what happens:</p>
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">That is a nice jump in performance!</p>
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">Using Memcached::Fast was kind of a mixed bag when looking at the benchmarks for mixing MySQL and Memcached in my tests:</p>
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">In this case I think Fast was slower when working with MySQL with a 256m BP because the slower returns from memcached acted as a bottleneck to thin the demands on MySQL to write data, smoothing out the work load.  When we eliminate this bottleneck with the Fast api, MySQL gets hammered.</p>
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">A couple of good things to remember here:  #1 resolving 1 bottleneck can open another bottleneck that is much worse.  #2  is to understand that not all API's are created equal.  Additionally the configuration and setup that works well on one system may not work well on another.  Because of this people often leave lots of performance on the table.  Don't just trust that your current API or config is optimal, test and make sure it fits your application.</p>
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">So adding Memcached on top of MySQL for our test application can significantly boost performance. But you notice that if we were running 100% in memcached and could cut out MySQL we could get 2.5x more performance.  As the number of writes against the database increase this gap will increase.  So let's ditch the database!  But wait!  you need the DB for  persistence, right?</p>
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">It depends.  A database may not be the best fit for every application.  There are several “NOSQL”  solutions out in the open source space that can give you some of the ease of a Memcached but with persistence most people use their database for.   Each application is different and understanding the application's requirements is key to picking an appropriate solution.   I am going to look at several database alternatives over the next few months.  I need to start somewhere, so I decided to start with Tokyo Tyrant and Cabinet.</p>
</div>
    <hr noshade style="margin:0;height:1px" />
    <p>Entry posted by matt |
      <a href="http://www.mysqlperformanceblog.com/2009/10/16/mysql_memcached_tyrant_part2/#comments">7 comments</a></p>
    <p>Add to: <a href="http://del.icio.us/post?url=http://www.mysqlperformanceblog.com/2009/10/16/mysql_memcached_tyrant_part2/&amp;title=MySQL-Memcached or NOSQL Tokyo Tyrant &#8211; part 2" title="Bookmark this post on del.icio.us"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/delicious.png" alt="delicious" /></a> | <a href="http://digg.com/submit?phase=2&amp;url=http://www.mysqlperformanceblog.com/2009/10/16/mysql_memcached_tyrant_part2/&amp;title=MySQL-Memcached or NOSQL Tokyo Tyrant &#8211; part 2" title="Digg this post on Digg.com"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/digg.png" alt="digg" /></a> | <a href="http://reddit.com/submit?url=http://www.mysqlperformanceblog.com/2009/10/16/mysql_memcached_tyrant_part2/&amp;title=MySQL-Memcached or NOSQL Tokyo Tyrant &#8211; part 2" title="Submit this post on reddit.com"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/reddit.png" alt="reddit" /></a> | <a href="http://www.netscape.com/submit/?U=http://www.mysqlperformanceblog.com/2009/10/16/mysql_memcached_tyrant_part2/&amp;T=MySQL-Memcached or NOSQL Tokyo Tyrant &#8211; part 2" title="Vote for this article on Netscape"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/netscape.gif" alt="netscape" /></a> | <a href="http://www.google.com/bookmarks/mark?op=add&amp;bkmk=http://www.mysqlperformanceblog.com/2009/10/16/mysql_memcached_tyrant_part2/&amp;title=MySQL-Memcached or NOSQL Tokyo Tyrant &#8211; part 2" title="Add to Google Bookmarks"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/google.png" alt="Google Bookmarks" /></a></p>]]></content:encoded>
			<wfw:commentRss>http://www.mysqlperformanceblog.com/2009/10/16/mysql_memcached_tyrant_part2/feed/</wfw:commentRss>
		<slash:comments>7</slash:comments>
		</item>
		<item>
		<title>MySQL-Memcached or NOSQL Tokyo Tyrant &#8211; part 1</title>
		<link>http://www.mysqlperformanceblog.com/2009/10/15/mysql-memcached-or-nosql-tokyo-tyrant-part-1/</link>
		<comments>http://www.mysqlperformanceblog.com/2009/10/15/mysql-memcached-or-nosql-tokyo-tyrant-part-1/#comments</comments>
		<pubDate>Thu, 15 Oct 2009 18:24:30 +0000</pubDate>
		<dc:creator>matt</dc:creator>
				<category><![CDATA[NOSQL]]></category>
		<category><![CDATA[benchmarks]]></category>
		<category><![CDATA[memcached]]></category>
		<category><![CDATA[mysql]]></category>
		<category><![CDATA[tips]]></category>

		<guid isPermaLink="false">http://www.mysqlperformanceblog.com/?p=1419</guid>
		<description><![CDATA[All to often people force themselves into using a database like MySQL with no thought into whether if its the best solution to there problem. Why?  Because their other applications use it, so why not the new application?  Over the past couple of months I have been doing a ton of work for clients who [...]]]></description>
			<content:encoded><![CDATA[<p>All to often people force themselves into using a database like MySQL with no thought into whether if its the best solution to there problem. Why?  Because their other applications use it, so why not the new application?  Over the past couple of months I have been doing a ton of work for clients who use their database like most people use memcached .  Lookup a row based on a key, update the data in the row, stuff the row back in the database.  Rinse and repeat.  Sure these setups vary sometimes, throwing in a “lookup” via username, or even the rare count.  But for the most part they are designed to be simple.</p>
<p>A classic example is a simple online game.  An online game may only require that an application retrieve a single record from the database.  The record may contain all the vital stats for the game, be updated and stuffed back into the database.  You would be surprised how many people use this type of system as I run into this type of application frequently.  Keeping it simple, ensures that application is generally mean and lean and performs well.  The issue is even this simple design can start to have issues as the data size increases and you blow through your available memory.  Is there a better architecture?  Is there a way to get more scalability out of your database?  Is the database even the best place for this data?</p>
<p>I decided to walk through setting up a very simple application that does what I have seen many clients do.  Using this application I can then compare using MySQL to using MySQL + Memcached, and then to other solutions like Tokyo Tyrant or Cassandra.   My Application does the following:</p>
<p>A.)  read a row from a database based on an integer based primary key<br />
B.)  Update data from that row and replace the stored contents on disk<br />
C.)  Use the data from that row to lookup up a row in another table based on a text field ( called email address ).</p>
<p>Seems simple enough right?  My two tables each contain 5M rows of data.  let's see what happens:</p>
<p><img class="size-full wp-image-1420" title="DB Fits into Memory" src="http://www.mysqlperformanceblog.com/wp-content/uploads/2009/10/db_right_option_html_1f7313bd.gif" alt="DB Fits into Memory" width="619" height="353" /></p>
<p><img class="size-full wp-image-1429" title="Chart of numbers" src="http://www.mysqlperformanceblog.com/wp-content/uploads/2009/10/db_right_option_html_m767bebf7.gif" alt="Chart of TPS for benchmark application" width="257" height="69" /></p>
<p>You can see a dramatic drop off in performance as my data falls out of memory, that's not cool is it?  After all database sizes tend to always grow and very rarely shrink.  Which leads to a challenge faced by almost everyone how do you maintain your performance in the face of increasing data size?</p>
<p>Here is where people start to scratch their heads.  They naturally assume they need to scale more, we need more memory!   If performance sucks, we must need more.  So here comes the bigger boxes, the read-only slaves,  the complex sharding systems, the discussions on cluster, more memcached.  We need to cover up the databases inefficiencies to ensure that our application scales.</p>
<p>The problem is for some applications, we are fixing symptoms, not the problem itself.  No matter how much you want it to fit,  some things may not work (like the Godfather 3).    The issue is people assume that data storage has to be in the database.  “It's data, it needs to go into the database.” is often the battle cry.   But hold on to your hat,  I  am going to shock you.  For some applications, putting your data in the database is silly.  Yes the guy who blogs on bigdbahead.com and is writing this on the mysqlperformanceblog is saying you may not want to use a database.  Heresy I know!  But for many of us we already accept storing data ( at least temporarily ) outside the DB.  Think memcached.</p>
<p>Almost everyone loves memcached, it's simple, fast, and just works.  When your dataset exceeds your memory limitations or the database can simply not keep up any more this solution can really boost performance.  I know you're thinking my simple key lookup should really benefit from memcached. So let's try it!  I took the simple app I created that reads two rows, and update one of them to read from memcached if available, remove on update, and read from the db only when required.  I tested with a memcached size of 1GB, 2GB, and 4GB.  For these tests I left Innodb with a 256M buffer pool, or roughly with 9% of the total data in memory.</p>
<p>let's look at the 1GB Setting:</p>
<div class="mceTemp mceIEcenter">
<dl id="attachment_1425" class="wp-caption aligncenter" style="width: 636px;">
<dt class="wp-caption-dt"><img class="size-full wp-image-1425" title="Ensure you have enough memory " src="http://www.mysqlperformanceblog.com/wp-content/uploads/2009/10/db_right_option_html_m5daa4d5f.gif" alt="Ensure you have enough memory for memcached" width="626" height="320" /> </dt>
</dl>
</div>
<p>What, a performance regression?  But we threw more memory at it!!   How can that be!</p>
<p>Memcached is not a cure all.  I have talked to many client's who say “we will just throw memcached at it”.   Sometimes an app will scream other times it won't... and yet others require lots and lots of memory allocated to memcached to be successful.    This application selects a random # between 1 and 2 Million and looks up the result via that key.  It then uses data from that random row to look up a second piece of information via email address.  Because the entire dataset  is about 4GB and only 1G is in memcached, I keep pushing data out of memcached to make room for new records I am reading from the database. Remember memcached needs repeatability to be helpful.   I am still getting a really solid # of hits in memcached, but the # of writes in MySQL coupled with the still large # of reads takes its toll.  Another place where I have seen this kill clients is in apps that do some sequential scanning and do not have enough memory for memcached.  For instance, if you have 1,000,000 rows of data, but enough memory to only store 500,000 rows... sequentially accessing this data will destroy the use of cache:</p>
<p>get record 1, miss, read from disk, cache record 1<br />
….<br />
get record 500,001, miss, read from disk, expunge record 1, cache record 500,001<br />
....<br />
get record 1, miss, read from disk, expunge record 500,001, cache record 1</p>
<p>you keep overwriting the cache before you can use it.  So here the complexity of adding memcached hurts us, because the cache is not actually buying us anything.</p>
<p>Now bumping this up to 2GB actually makes the TPS jump around a lot, sometimes hitting 400 or 500 TPS and other times hitting as high as 1800 TPS.  My guess is the movement is caused by the random #'s being generated and simply the luck of the draw.</p>
<p>Finally let's look when we have 4GB of memory allocated to memcached (full dataset fits ):</p>
<p><img class="size-full wp-image-1423" title="Transactions with and without Memcached" src="http://www.mysqlperformanceblog.com/wp-content/uploads/2009/10/db_right_option_html_m2dc4c2b4.gif" alt="Transactions with and without Memcached" width="625" height="331" /></p>
<p>Here you can see that our “transactions”  per second for this app increased almost 10Xby using memcached.  The TPS I get here vary from 1100 TPS to just under 2000TPS with the average around 1400TPS.   I think we would all be very happy if we could get a 10X performance boost from your application.</p>
<p>But wouldn't it be great if we could get more?  I mean our reads are going pretty fast, but our writes leave a lot to be desired:</p>
<div class="mceTemp mceIEcenter"><img class="size-full wp-image-1424" title="Read -vs- write times " src="http://www.mysqlperformanceblog.com/wp-content/uploads/2009/10/db_right_option_html_m4d18b08e.gif" alt="Read -vs- write times with memcached + mysql mixed" width="648" height="286" /></div>
<p>Over 17 MS to do an update.  Wouldn't be great to just eliminate all the updates as well?  What sort of throughput would we get?   I will show you in part 2.  Part 2 of this post will talk about performance in a 100% pure memcached environment. Part 3 will focus on these same benchmarks in Tokyo tyrant.</p>
<div id="_mcePaste" style="overflow: hidden; position: absolute; left: -10000px; top: 0px; width: 1px; height: 1px;"><!-- 		@page { margin: 0.79in } 		P { margin-bottom: 0.08in } --></p>
<p style="margin-bottom: 0in;">All to often people force themselves into using a database like MySQL with no thought into whether if its the best solution to there problem. Why?  Because their other applications use it, so why not the new application?  Over the past couple of months I have been doing a ton of work for clients who use their database like most people use memcached .  Lookup a row based on a key, update the data in the row, stuff the row back in the database.  Rinse and repeat.  Sure these setups vary sometimes, throwing in a “lookup” via username, or even the rare count.  But for the most part they are designed to be simple.</p>
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">A classic example is a simple online game.  An online game may only require that an application retrieve a single record from the database.  The record may contain all the vital stats for the game, be updated and stuffed back into the database.  You would be surprised how many people use this type of system as I run into this type of application frequently.  Keeping it simple, ensures that application is generally mean and lean and performs well.  The issue is even this simple design can start to have issues as the data size increases and you blow through your available memory.  Is there a better architecture?  Is there a way to get more scalability out of your database?  Is the database even the best place for this data?</p>
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">I decided to walk through setting up a very simple application that does what I have seen many clients do.  Using this application I can then compare using MySQL to using MySQL + Memcached, and then to other solutions like Tokyo Tyrant or Cassandra.   My Application does the following:</p>
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">A.)  read a row from a database based on an integer based primary key</p>
<p style="margin-bottom: 0in;">B.)  Update data from that row and replace the stored contents on disk</p>
<p style="margin-bottom: 0in;">C.)  Use the data from that row to lookup up a row in another table based on a text field ( called email address ).</p>
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">Seems simple enough right?  My two tables each contain 5M rows of data.  let's see what happens:</p>
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">You can see a dramatic drop off in performance as my data falls out of memory, that's not cool is it?  After all database sizes tend to always grow and very rarely shrink.  Which leads to a challenge faced by almost everyone how do you maintain your performance in the face of increasing data size?</p>
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">Here is where people start to scratch their heads.  They naturally assume they need to scale more, we need more memory!   If performance sucks, we must need more.  So here comes the bigger boxes, the read-only slaves,  the complex sharding systems, the discussions on cluster, more memcached.  We need to cover up the databases inefficiencies to ensure that our application scales.</p>
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">The problem is for some applications, we are fixing symptoms, not the problem itself.  No matter how much you want it to fit,  somethings may not work (like the Godfather 3).    The issue is people assume that data storage has to be in the database.  “It's data, it needs to go into the database.” is often the battle cry.   But hold on to your hat,  I  am going to shock you.  For some applications, putting your data in the database is silly.  Yes the guy who blogs on bigdbahead.com and is writing this on the mysqlperformanceblog is saying you may not want to use a database.  Heresy I know!  But for many of us we already accept storing data ( at least temporarily ) outside the DB.  Think memcached.</p>
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">Almost everyone loves memcached, it's simple, fast, and just works.  When your dataset exceeds your memory limitations or the database can simply not keep up any more this solution can really boost performance.  I know you're thinking my simple key lookup should really benefit from memcached. So let's try it!  I took the simple app I created that reads two rows, and update one of them to read from memcached if available, remove on update, and read from the db only when required.  I tested with a memcached size of 1GB, 2GB, and 4GB.  For these tests I left Innodb with a 256M buffer pool, or roughly with 9% of the total data in memory.</p>
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">let's look at the 1GB Setting:</p>
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">What, a performance regression?  But we threw more memory at it!!   How can that be!</p>
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">Memcached is not a cure all.  I have talked to many client's who say “we will just throw memcached as it”.   Sometimes an app will scream other times it won't... and yet others require lots and lots of memory allocated to memcached to be successful.    This application selects a random # between 1 and 2 Million and looks up the result via that key.  It then uses data from that random row to look up a second piece of information via email address.  Because the entire dataset  is about 4GB and only 1G is in memcached, I keep pushing data out of memcached to make room for new records I am reading from the database. Remember memcached needs repeatability to be helpful.   I am still getting a really solid # of hits in memcached, but the # of writes in MySQL coupled with the still large # of reads takes its toll.  Another place where I have seen this kill clients is in apps that do some sequential scanning and do not have enough memory for memcached.  For instance, if you have 1,000,000 rows of data, but enough memory to only store 500,000 rows... sequentially accessing this data will destroy the use of cache:</p>
<p style="margin-bottom: 0in;">
<ul>
<li>
<p style="margin-bottom: 0in;">get record 1, miss, read from 	disk, cache record 1</p>
</li>
<li>
<p style="margin-bottom: 0in;">….</p>
</li>
<li>
<p style="margin-bottom: 0in;">get record 500,001, miss, read 	from disk, expunge record 1, cache record 500,001</p>
</li>
<li>
<p style="margin-bottom: 0in;">....</p>
</li>
<li>
<p style="margin-bottom: 0in;">get record 1, miss, read from 	disk, expunge record 500,001, cache record 1</p>
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">you keep overwriting the cache before 	you can use it.  So here the complexity of adding memcached hurts 	us, because the cache is not actually buying us anything.</p>
</li>
</ul>
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">Now bumping this up to 2GB actually makes the TPS jump around a lot, sometimes hitting 400 or 500 TPS and other times hitting as high as 1800 TPS.  My guess is the movement is caused by the random #'s being generated and simply the luck of the draw.</p>
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">Finally let's look when we have 4GB of memory allocated to memcached (full dataset fits ):</p>
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">Here you can see that our “transactions”  per second for this app increased almost 10Xby using memcached.  The TPS I get here vary from 1100 TPS to just under 2000TPS with the average around 1400TPS.   I think we would all be very happy if we could get a 10X performance boost from your application.</p>
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">But wouldn't it be great if we could get more?  I mean our reads are going pretty fast, but our writes leave a lot to be desired:</p>
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">
<p style="margin-bottom: 0in;">Over 17 MS to do an update.  Wouldn't be great to just eliminate all the updates as well?  What sort of throughput would we get?</p>
</div>
    <hr noshade style="margin:0;height:1px" />
    <p>Entry posted by matt |
      <a href="http://www.mysqlperformanceblog.com/2009/10/15/mysql-memcached-or-nosql-tokyo-tyrant-part-1/#comments">13 comments</a></p>
    <p>Add to: <a href="http://del.icio.us/post?url=http://www.mysqlperformanceblog.com/2009/10/15/mysql-memcached-or-nosql-tokyo-tyrant-part-1/&amp;title=MySQL-Memcached or NOSQL Tokyo Tyrant &#8211; part 1" title="Bookmark this post on del.icio.us"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/delicious.png" alt="delicious" /></a> | <a href="http://digg.com/submit?phase=2&amp;url=http://www.mysqlperformanceblog.com/2009/10/15/mysql-memcached-or-nosql-tokyo-tyrant-part-1/&amp;title=MySQL-Memcached or NOSQL Tokyo Tyrant &#8211; part 1" title="Digg this post on Digg.com"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/digg.png" alt="digg" /></a> | <a href="http://reddit.com/submit?url=http://www.mysqlperformanceblog.com/2009/10/15/mysql-memcached-or-nosql-tokyo-tyrant-part-1/&amp;title=MySQL-Memcached or NOSQL Tokyo Tyrant &#8211; part 1" title="Submit this post on reddit.com"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/reddit.png" alt="reddit" /></a> | <a href="http://www.netscape.com/submit/?U=http://www.mysqlperformanceblog.com/2009/10/15/mysql-memcached-or-nosql-tokyo-tyrant-part-1/&amp;T=MySQL-Memcached or NOSQL Tokyo Tyrant &#8211; part 1" title="Vote for this article on Netscape"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/netscape.gif" alt="netscape" /></a> | <a href="http://www.google.com/bookmarks/mark?op=add&amp;bkmk=http://www.mysqlperformanceblog.com/2009/10/15/mysql-memcached-or-nosql-tokyo-tyrant-part-1/&amp;title=MySQL-Memcached or NOSQL Tokyo Tyrant &#8211; part 1" title="Add to Google Bookmarks"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/google.png" alt="Google Bookmarks" /></a></p>]]></content:encoded>
			<wfw:commentRss>http://www.mysqlperformanceblog.com/2009/10/15/mysql-memcached-or-nosql-tokyo-tyrant-part-1/feed/</wfw:commentRss>
		<slash:comments>13</slash:comments>
		</item>
		<item>
		<title>How to generate per-database traffic statistics using mk-query-digest</title>
		<link>http://www.mysqlperformanceblog.com/2009/09/16/how-to-generate-per-database-traffic-statistics-using-mk-query-digest/</link>
		<comments>http://www.mysqlperformanceblog.com/2009/09/16/how-to-generate-per-database-traffic-statistics-using-mk-query-digest/#comments</comments>
		<pubDate>Wed, 16 Sep 2009 18:47:42 +0000</pubDate>
		<dc:creator>Ryan Lowe</dc:creator>
				<category><![CDATA[tips]]></category>
		<category><![CDATA[tools]]></category>
		<category><![CDATA[Maatkit]]></category>
		<category><![CDATA[sharding]]></category>

		<guid isPermaLink="false">http://www.mysqlperformanceblog.com/?p=1162</guid>
		<description><![CDATA[We often encounter customers who have partitioned their applications among a number of databases within the same instance of MySQL (think application service providers who have a separate database per customer organization ... or wordpress-mu type of apps).  For example, take the following single MySQL instance with multiple (identical) databases:

SHOW DATABASES;
+----------+
&#124; Database &#124;
+----------+
&#124; db1 [...]]]></description>
			<content:encoded><![CDATA[<p>We often encounter customers who have partitioned their applications among a number of databases within the same instance of MySQL (think application service providers who have a separate database per customer organization ... or wordpress-mu type of apps).  For example, take the following single MySQL instance with multiple (identical) databases:</p>
<pre>
SHOW DATABASES;
+----------+
| Database |
+----------+
| db1      |
| db2      |
| db3      |
| db4      |
| mysql    |
+----------+
</pre>
<p>Separating the data in this manner is a great setup for being able to scale by simply migrating a subset of the databases to a different physical host when the existing host begins to get overloaded.  But MySQL doesn't allow us to examine statistics on a per-database basis.  </p>
<p>Enter Maatkit.</p>
<p>There is an often-ignored gem in <a href="http://maatkit.org/">Maatkit</a>'s mk-query-digest, and that is the --group-by argument.  This can be used to aggregate information by tables, hosts, users, or databases (full documentation is available via perldoc).</p>
<pre>
%> perl mk-query-digest --limit 100% --group-by db slow.log
...
# Rank Query ID Response time Calls R/Call Item
# ==== ======== ============= ===== ====== ====
#    1 0x       6000 60.0%    6000  0.5124 db3
#    2 0x       2000 20.0%    2000  0.0112 db1
#    3 0x       1500 15.0%    1500  0.1665 db2
#    4 0x        500  5.0%     500  0.0022 db4
</pre>
<p>So here, we can see that the majority (60%, to be exact) of execution time is spent in db3.  If the server is reaching it's capacity and the next most useful performance optimization is to migrate a database to a different server, you know exactly which database to move (db3) and how much room that will give you on the original host (60% growth) and on the new host (40% growth), which may have a direct bearing on your hardware selection.</p>
<p>Let Baron know how awesome you think this is by getting him a gift from his <a href="http://www.amazon.com/wishlist/LOE4ZUTKFU39">Amazon Wish List</a>!</p>
    <hr noshade style="margin:0;height:1px" />
    <p>Entry posted by Ryan Lowe |
      <a href="http://www.mysqlperformanceblog.com/2009/09/16/how-to-generate-per-database-traffic-statistics-using-mk-query-digest/#comments">No comment</a></p>
    <p>Add to: <a href="http://del.icio.us/post?url=http://www.mysqlperformanceblog.com/2009/09/16/how-to-generate-per-database-traffic-statistics-using-mk-query-digest/&amp;title=How to generate per-database traffic statistics using mk-query-digest" title="Bookmark this post on del.icio.us"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/delicious.png" alt="delicious" /></a> | <a href="http://digg.com/submit?phase=2&amp;url=http://www.mysqlperformanceblog.com/2009/09/16/how-to-generate-per-database-traffic-statistics-using-mk-query-digest/&amp;title=How to generate per-database traffic statistics using mk-query-digest" title="Digg this post on Digg.com"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/digg.png" alt="digg" /></a> | <a href="http://reddit.com/submit?url=http://www.mysqlperformanceblog.com/2009/09/16/how-to-generate-per-database-traffic-statistics-using-mk-query-digest/&amp;title=How to generate per-database traffic statistics using mk-query-digest" title="Submit this post on reddit.com"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/reddit.png" alt="reddit" /></a> | <a href="http://www.netscape.com/submit/?U=http://www.mysqlperformanceblog.com/2009/09/16/how-to-generate-per-database-traffic-statistics-using-mk-query-digest/&amp;T=How to generate per-database traffic statistics using mk-query-digest" title="Vote for this article on Netscape"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/netscape.gif" alt="netscape" /></a> | <a href="http://www.google.com/bookmarks/mark?op=add&amp;bkmk=http://www.mysqlperformanceblog.com/2009/09/16/how-to-generate-per-database-traffic-statistics-using-mk-query-digest/&amp;title=How to generate per-database traffic statistics using mk-query-digest" title="Add to Google Bookmarks"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/google.png" alt="Google Bookmarks" /></a></p>]]></content:encoded>
			<wfw:commentRss>http://www.mysqlperformanceblog.com/2009/09/16/how-to-generate-per-database-traffic-statistics-using-mk-query-digest/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Which adaptive should we use?</title>
		<link>http://www.mysqlperformanceblog.com/2009/09/15/which-adaptive-should-we-use/</link>
		<comments>http://www.mysqlperformanceblog.com/2009/09/15/which-adaptive-should-we-use/#comments</comments>
		<pubDate>Wed, 16 Sep 2009 00:12:44 +0000</pubDate>
		<dc:creator>Yasufumi</dc:creator>
				<category><![CDATA[Innodb]]></category>
		<category><![CDATA[benchmarks]]></category>
		<category><![CDATA[tips]]></category>
		<category><![CDATA[tuning]]></category>
		<category><![CDATA[xtradb]]></category>

		<guid isPermaLink="false">http://www.mysqlperformanceblog.com/?p=1125</guid>
		<description><![CDATA[As you may know, InnoDB has 2 limits for unflushed modified blocks in the buffer pool. The one is from physical size of the buffer pool. And the another one is oldness of the block which is from the capacity of transaction log files.
In the case of heavy updating workload, the modified ages of the [...]]]></description>
			<content:encoded><![CDATA[<p>As you may know, InnoDB has 2 limits for unflushed modified blocks in the buffer pool. The one is from physical size of the buffer pool. And the another one is oldness of the block which is from the capacity of transaction log files.</p>
<p>In the case of heavy updating workload, the modified ages of the many blocks are clustered. And to reduce the maximum of the modified ages InnoDB needs to flush many of the blocks in a short time, if these are not flushed at all. Then the flushing storm affect the performance seriously.</p>
<p>We suggested the "adaptive_checkpoint" option of constant flushing to avoid such a flushing storm. And finally, the newest InnoDB Plugin 1.0.4 has the new similar option "adaptive_flushing" as native.</p>
<p>Let's check the adaptive flushing options at this post.</p>
<p><em><strong>HOW THEY WORK</strong></em></p>
<p><strong>&lt; adaptive_checkpoint=reflex (older method)&gt;</strong></p>
<table border="1" cellspacing="0" cellpadding="2">
<tbody>
<tr>
<td>oldest modified age %/max</td>
<td>behavior</td>
</tr>
<tr>
<td>0 ~ 50%</td>
<td>nothing</td>
</tr>
<tr>
<td>50% ~ 75%</td>
<td>constant flushing (weak)</td>
</tr>
<tr>
<td>75% ~ 87.5%</td>
<td>constant flushing (strong)</td>
</tr>
<tr>
<td>87.5% ~</td>
<td>(flushing storm)</td>
</tr>
</tbody>
</table>
<p><strong>&lt; adaptive_checkpoint=estimate (newer method)&gt;</strong></p>
<table border="1" cellspacing="0" cellpadding="2">
<tbody>
<tr>
<td>oldest modified age %/max</td>
<td>behavior</td>
</tr>
<tr>
<td>0 ~ 50%</td>
<td>nothing</td>
</tr>
<tr>
<td>50% ~ 87.5%</td>
<td>estimate flushing as bellow *</td>
</tr>
<tr>
<td>87.5% ~</td>
<td>(flushing storm)</td>
</tr>
</tbody>
</table>
<p>* estimate blocks to flush based on...</p>
<ul>
<li> how many modified and unflushed blocks</li>
<li> progress speed of the transaction log</li>
<li> the modified age average of all blocks to flush</li>
</ul>
<p><strong>&lt; adaptive_flushing (default "true" at 1.0.4)&gt;</strong><br />
Its behavior is not based on the oldest modified age %.</p>
<p>* the how many blocks to flush based on...</p>
<ul>
<li> how many modified and unflushed blocks</li>
<li> progress speed of the transaction log</li>
<li> transaction log capacity</li>
</ul>
<p>(and adjust along with another flushing)</p>
<p>And it doesn't exceed the equivalent to "constant flushing (strong)"<br />
of the "adaptive_checkpoint=reflex"</p>
<p><em><strong>RESULTS</strong></em></p>
<p>TPC-C like workload (100WH: 16 session full)</p>
<p>innodb_buffer_pool_size = 16G<br />
innodb_max_dirty_pages_pct = 90</p>
<p>innodb_log_file_size = 512M<br />
innodb_log_files_in_group = 2</p>
<p>innodb_io_capacity = 4000<br />
innodb_read_io_threads = 8<br />
innodb_write_io_threads = 8<br />
innodb_flush_method = O_DIRECT</p>
<p>innodb_thread_concurrency = 0<br />
innodb_ibuf_active_contract = 1</p>
<p><strong>&lt;none&gt;</strong><br />
innodb_adaptive_flushing = false<br />
innodb_adaptive_checkpoint = none</p>
<p><strong>&lt;flushing&gt;</strong><br />
innodb_adaptive_flushing = true</p>
<p><strong>&lt;reflex&gt;</strong><br />
innodb_adaptive_flushing = false<br />
innodb_adaptive_checkpoint = reflex</p>
<p><strong>&lt;estimate&gt;</strong><br />
innodb_adaptive_flushing = false<br />
innodb_adaptive_checkpoint = estimate</p>
<p><strong><em>case1: "innodb_doublewrite = false"</em></strong></p>
<p>[0~1800 sec.]</p>
<p><img class="alignnone size-full wp-image-1132" title="no_doublewrite-0-1800" src="http://www.mysqlperformanceblog.com/wp-content/uploads/2009/09/no_doublewrite-0-18001.png" alt="no_doublewrite-0-1800" width="705" height="388" /></p>
<p>[1200~1800 sec.]</p>
<p><img class="alignnone size-full wp-image-1134" title="no_doublewrite-1200-1800" src="http://www.mysqlperformanceblog.com/wp-content/uploads/2009/09/no_doublewrite-1200-1800.png" alt="no_doublewrite-1200-1800" width="705" height="388" /></p>
<p>averages (1200~1800 sec.)<br />
none:     6868.92<br />
flushing: 6655.92<br />
reflex:     6481<br />
estimate: 6575.88</p>
<p><strong><em>case2: "innodb_doublewrite = true"</em></strong></p>
<p>[0~1800 sec.]</p>
<p><img class="alignnone size-full wp-image-1137" title="doublewrite-0-1800" src="http://www.mysqlperformanceblog.com/wp-content/uploads/2009/09/doublewrite-0-1800.png" alt="doublewrite-0-1800" width="705" height="388" /></p>
<p>[1200~1800 sec.]</p>
<p><img class="alignnone size-full wp-image-1138" title="doublewrite-1200-1800" src="http://www.mysqlperformanceblog.com/wp-content/uploads/2009/09/doublewrite-1200-1800.png" alt="doublewrite-1200-1800" width="705" height="388" /></p>
<p>averages (1200~1800 sec.)<br />
none:     6569.48<br />
flushing: 5090.12<br />
reflex:     6871.9<br />
estimate: 6609.9</p>
<p><strong><em>CONSIDERINGS</em></strong></p>
<p>The new "adaptive_flushing" seems to be tuned for "innodb_doublewrite = false" only.<br />
(cause too much flushing for "innodb_doublewrite = true")</p>
<p>"innodb_adaptive_checkpoint = reflex" and "adaptive_flushing" need tuning innodb_io_capacity properly.<br />
(The result is based on proper value "innodb_io_capacity = 4000")</p>
<p>"innodb_adaptive_checkpoint = estimate" is not depend on innodb_io_capacity, and it seems more "soft" result than the other methods.</p>
<p>So,</p>
<p>The "adaptive_flushing" seems good when we use "innodb_doublewrite = false".</p>
<p>"innodb_adaptive_checkpoint = estimate" may be safe for "innodb_doublewrite = true" for now.</p>
<p>I should adjust "adaptive_flushing" for "innodb_doublewrite = true". <img src='http://www.mysqlperformanceblog.com/wp-includes/images/smilies/icon_smile.gif' alt=':-)' class='wp-smiley' /> </p>
<hr /><em><strong>IMPROVEMENTS</strong></em> (16 Sep.)</p>
<p>I have tried to improve the performance. But I couldn't adjust performance of adaptive_flushing with doublewrite... So, instead of that, estimate is adjusted more.</p>
<p><strong><em>case1: "innodb_doublewrite = false"</em></strong></p>
<p>[0~1800 sec.]</p>
<p><img class="alignnone size-full wp-image-1159" title="no_doublewrite-0-1800_2" src="http://www.mysqlperformanceblog.com/wp-content/uploads/2009/09/no_doublewrite-0-1800_21.png" alt="no_doublewrite-0-1800_2" width="705" height="388" /></p>
<p><strong><em>case2: "innodb_doublewrite = true"</em></strong></p>
<p>[0~1800 sec.]</p>
<p><img class="alignnone size-full wp-image-1160" title="doublewrite-0-1800_2" src="http://www.mysqlperformanceblog.com/wp-content/uploads/2009/09/doublewrite-0-1800_2.png" alt="doublewrite-0-1800_2" width="705" height="388" /></p>
<p>The new "innodb_adaptive_checkpoint = estimate" seems to be more stable than before. The next question is, <strong>"Why is estimate + doublewrite better than the other no doublewrite settings?"</strong>... <img src='http://www.mysqlperformanceblog.com/wp-includes/images/smilies/icon_smile.gif' alt=':-)' class='wp-smiley' /> </p>
<hr /><strong>ADDITIONAL TEST IN ANOTHER BALANCE</strong></p>
<p>Also tested in another configuration to check the behaviors in another balance of IO bound.</p>
<p>&lt;changes&gt;<br />
innodb_buffer_pool_size = 2G   (1/8)<br />
innodb_log_file_size = 128M    (1/4)</p>
<p><strong><em>case1: "innodb_doublewrite = false"</em></strong></p>
<p>[0~1200 sec.]</p>
<p><img class="alignnone size-full wp-image-1202" title="no_doublewrite_2-0-1200" src="http://www.mysqlperformanceblog.com/wp-content/uploads/2009/09/no_doublewrite_2-0-1200.png" alt="no_doublewrite_2-0-1200" width="705" height="387" /></p>
<p><strong><em>case2: "innodb_doublewrite = true"</em></strong></p>
<p>[0~1200 sec.]</p>
<p><img class="alignnone size-full wp-image-1203" title="doublewrite_2-0-1200" src="http://www.mysqlperformanceblog.com/wp-content/uploads/2009/09/doublewrite_2-0-1200.png" alt="doublewrite_2-0-1200" width="705" height="387" /></p>
<p>"innodb_adaptive_checkpoint = estimate" seems stable but seems to flush much in its initial phase. The adaptive_flushing seems to be more soft in this cases.</p>
<p>Hmmm....</p>
<p><strong>We should choose suitable adaptive_xxx for each workload for now</strong>... <img src='http://www.mysqlperformanceblog.com/wp-includes/images/smilies/icon_smile.gif' alt=':-)' class='wp-smiley' /> </p>
    <hr noshade style="margin:0;height:1px" />
    <p>Entry posted by Yasufumi |
      <a href="http://www.mysqlperformanceblog.com/2009/09/15/which-adaptive-should-we-use/#comments">12 comments</a></p>
    <p>Add to: <a href="http://del.icio.us/post?url=http://www.mysqlperformanceblog.com/2009/09/15/which-adaptive-should-we-use/&amp;title=Which adaptive should we use?" title="Bookmark this post on del.icio.us"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/delicious.png" alt="delicious" /></a> | <a href="http://digg.com/submit?phase=2&amp;url=http://www.mysqlperformanceblog.com/2009/09/15/which-adaptive-should-we-use/&amp;title=Which adaptive should we use?" title="Digg this post on Digg.com"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/digg.png" alt="digg" /></a> | <a href="http://reddit.com/submit?url=http://www.mysqlperformanceblog.com/2009/09/15/which-adaptive-should-we-use/&amp;title=Which adaptive should we use?" title="Submit this post on reddit.com"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/reddit.png" alt="reddit" /></a> | <a href="http://www.netscape.com/submit/?U=http://www.mysqlperformanceblog.com/2009/09/15/which-adaptive-should-we-use/&amp;T=Which adaptive should we use?" title="Vote for this article on Netscape"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/netscape.gif" alt="netscape" /></a> | <a href="http://www.google.com/bookmarks/mark?op=add&amp;bkmk=http://www.mysqlperformanceblog.com/2009/09/15/which-adaptive-should-we-use/&amp;title=Which adaptive should we use?" title="Add to Google Bookmarks"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/google.png" alt="Google Bookmarks" /></a></p>]]></content:encoded>
			<wfw:commentRss>http://www.mysqlperformanceblog.com/2009/09/15/which-adaptive-should-we-use/feed/</wfw:commentRss>
		<slash:comments>12</slash:comments>
		</item>
		<item>
		<title>3 ways MySQL uses indexes</title>
		<link>http://www.mysqlperformanceblog.com/2009/09/12/3-ways-mysql-uses-indexes/</link>
		<comments>http://www.mysqlperformanceblog.com/2009/09/12/3-ways-mysql-uses-indexes/#comments</comments>
		<pubDate>Sat, 12 Sep 2009 15:31:32 +0000</pubDate>
		<dc:creator>peter</dc:creator>
				<category><![CDATA[tips]]></category>

		<guid isPermaLink="false">http://www.mysqlperformanceblog.com/?p=1097</guid>
		<description><![CDATA[I often see people confuse different ways MySQL can use indexing, getting wrong ideas on what query performance they should expect.   There are 3 main ways how MySQL can use the indexes for query execution, which are not mutually exclusive, in fact some queries will use indexes for all 3 purposes listed here. [...]]]></description>
			<content:encoded><![CDATA[<p>I often see people confuse different ways MySQL can use indexing, getting wrong ideas on what query performance they should expect.   There are 3 main ways how MySQL can use the indexes for query execution, which are not mutually exclusive, in fact some queries will use indexes for all 3 purposes listed here. </p>
<p><strong>Using index to find rows</strong>  The main purpose of the index is to find rows quickly - without scanning whole data set. This is most typical reason index gets added on the first place.  Most popular index type in  MySQL - BTREE can speed up equality and prefix range matches.   So if you have index on <strong>(A,B)</strong>    This index can be used to lookup rows for WHERE clauses like  <strong>A=5  </strong> ;  <strong> A BETWEEN 5 AND 10</strong> ;  <strong>A=5 AND B BETWEEN 5 AND 10</strong>   it however will <strong>NOT</strong> be able to help lookup rows for  <strong>B BETWEEN 5 AND 10</strong> predicate because it is not index prefix.   It is important to look at <strong>key_len</strong> column in explain plan to see how many index parts are actually used for row lookup.  Very common problem I see is multi column indexes which are used but only to their short prefix which is not very selective.   A lot of this mistakes come from missing one very important MySQL limitation -  once MySQL runs into the interval range it will not use any further index parts.   If you have  <strong>A BETWEEN 5 AND 10 AND B=5</strong> for the same index MySQL will use the index... but it will only use <strong>A</strong> prefix for row lookups and scan whole <strong>A BETWEEN 5 AND 10</strong> range.    It  is interesting to note this limitation only applies to interval ranges - for enumerated ranges MySQL will use both key parts.   Hence if you change this predicate to   <strong>A IN (5,6,7,8,9,10) AND B=5</strong>  you will quite likely see improved query performance.  Beware however of large nested enumerated ranges they are very hard on the optimizer.    This just describes how MySQL uses single index - there are more complex rules of how indexes will be used if you look at multiple indexes usage with "index merge"</p>
<p><strong>Using Index to Sort Data</strong>  Another great benefit of BTREE index is - it allows to retrieve data in sorted form hence avoiding external sort process for executing of queries which require sorting.  Using index for sorting often comes together with using index to find rows, however it can also be used just for sort for example if you're just using ORDER BY without and where clauses on the table.  In such case you would see "Index" type in explain which correspond to scanning (potentially) complete table in the index order.   It is very important to understand in which conditions index can be used to sort data together with restricting amount of rows.  Looking at the same index <strong>(A,B) </strong>    things like   <strong>ORDER BY A</strong>   ;  <strong>ORDER BY A,B</strong> ;  <strong> ORDER BY A DESC, B DESC</strong> will be able to use full index for sorting (note MySQL may not select to use index for sort if you sort full table without a limit).   However <strong>ORDER BY B</strong> or <strong>ORDER BY A, B DESC </strong> will not be able to use index because requested order does not line up with the order of data in BTREE.     If you have both restriction and sorting things like this would work  <strong>A=5 ORDER BY B</strong>  ; <strong>A=5 ORDER BY B DESC;</strong>    <strong>A>5 ORDER BY A</strong> ;  <strong>A>5 ORDER BY A,B</strong> ; <strong>A>5 ORDER BY A DESC</strong>    which again can be easily visualized as scanning a range in BTREE.   Things like this however would not work  <strong>A>5 ORDER BY B</strong>  ,   <strong>A>5 ORDER BY A,B  DESC</strong>  or  <strong>A IN (3,4) ORDER BY B</strong> - in these cases getting data in sorting form would require a bit more than simple range scan in the BTREE and MySQL decides to pass it on.   There are some <a href="http://www.mysqlperformanceblog.com/2006/08/14/mysql-followup-on-union-for-query-optimization-query-profiling/">workarounds</a> you can use though.</p>
<p><strong>Using index to read data </strong>  Some storage engines (MyISAM and Innodb included)  can also use index to read the data, hence avoiding to read the row data itself.  This is not simply savings of having 2 reads per index entry instead of one but it can save IO orders of magnitude in some cases - Indexes are sorted (at least on the page boundary) so  doing index range scan you typically get many index entries from the same page but the rows itself can be scattered across many pages requiring potentially a lot of IOs.  On  top of that if you just need access to couple of columns<br />
index can be simply much smaller than the data which is one of the reason covering indexes help to speed up queries even if data is in memory.     If MySQL is only reading index and not accessing rows you will see "using index" in EXPLAIN output.     </p>
<p>These are the main "core" use for indexes. You can also see others like using index for group by but I think they can be pretty much looked as one of these 3 ways described.</p>
    <hr noshade style="margin:0;height:1px" />
    <p>Entry posted by peter |
      <a href="http://www.mysqlperformanceblog.com/2009/09/12/3-ways-mysql-uses-indexes/#comments">14 comments</a></p>
    <p>Add to: <a href="http://del.icio.us/post?url=http://www.mysqlperformanceblog.com/2009/09/12/3-ways-mysql-uses-indexes/&amp;title=3 ways MySQL uses indexes" title="Bookmark this post on del.icio.us"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/delicious.png" alt="delicious" /></a> | <a href="http://digg.com/submit?phase=2&amp;url=http://www.mysqlperformanceblog.com/2009/09/12/3-ways-mysql-uses-indexes/&amp;title=3 ways MySQL uses indexes" title="Digg this post on Digg.com"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/digg.png" alt="digg" /></a> | <a href="http://reddit.com/submit?url=http://www.mysqlperformanceblog.com/2009/09/12/3-ways-mysql-uses-indexes/&amp;title=3 ways MySQL uses indexes" title="Submit this post on reddit.com"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/reddit.png" alt="reddit" /></a> | <a href="http://www.netscape.com/submit/?U=http://www.mysqlperformanceblog.com/2009/09/12/3-ways-mysql-uses-indexes/&amp;T=3 ways MySQL uses indexes" title="Vote for this article on Netscape"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/netscape.gif" alt="netscape" /></a> | <a href="http://www.google.com/bookmarks/mark?op=add&amp;bkmk=http://www.mysqlperformanceblog.com/2009/09/12/3-ways-mysql-uses-indexes/&amp;title=3 ways MySQL uses indexes" title="Add to Google Bookmarks"><img src="http://www.mysqlperformanceblog.com/wp-content/themes/boxy-but-gold/images/google.png" alt="Google Bookmarks" /></a></p>]]></content:encoded>
			<wfw:commentRss>http://www.mysqlperformanceblog.com/2009/09/12/3-ways-mysql-uses-indexes/feed/</wfw:commentRss>
		<slash:comments>14</slash:comments>
		</item>
	</channel>
</rss>
