I said in my last post, that we’re interested in speaking at MySQL meetups, and I’m happy to say that the Los Angeles MySQL Meetup has taken us up on the offer.

On November 18th, I’ll be giving an introductory talk on InnoDB/XtraDB Performance Optimization.  I will be the second speaker, with Carl Gelbart first speaking on Infobright.

What brings me to LA?  On the same day (18th Nov) I’ll be teaching a one day class on Performance Optimization for MySQL with InnoDB and XtraDB.  If you haven’t signed up yet – spaces are still available.

Entry posted by Morgan Tocker | No comment

Add to: | | | |

So long time after PPC 2009 there was available version mysql-galera-0.6, which had serious flow, to setup a new node you had to take down whole cluster. And all this time Codership ( company that develops Galera) was working on 0.7 release that introduces node propagation keeping cluster online. You can play with 0.7pre release by yourself MySQL/Galera Release 0.7pre.

In current version propagation is done by mysqldump from one of nodes (”donor”). In next release Codership is going to support LVM snapshot and xtrabackup which will make the setup of new node even easier. The current annoyance I see is that if you shutdown one node for short period of time for quick maintenance, after start, the node has to load whole mysqldump, like it is new empty node. I hope Codership guys will address this also.
Another thing I miss for now is support of InnoDB-plugin, which as we know performs much better than standard InnoDB ®.

So what is so interesting about Galera. Couple things:

- High Availability. Any of N standby nodes are available immediately when main node fails. Galera is serious pretender to be included to the list, Yves put recently, http://www.mysqlperformanceblog.com/2009/10/16/finding-your-mysql-high-availability-solution-%e2%80%93-the-questions/. I am not sure how many nines it will provide , but efforts on test setup and deployment should be comparable with MMM setup.

- Scale Writes. Galera allows to write to any of N nodes and automatically propagate to other nodes. It sounds too ideal, and there is drawback – with increasing amount of nodes you write to, your transaction rollback rate may increase, especially if you working on the same dataset. You can find some results on Codership’s page, and I am going to run my own benchmarks also. Also from benchmark you can see that communication overhead maybe significant for short writes.

- Scale Reads. It can be done with regular replication, but with synchronous your “slaves-nodes” are in the same state, there is no “slave behind”. When you read from any slave, you read actual data. Although it also has serious drawback – our cluster is fast as fast the “weakest” node in the chain. So if one node gets overloaded and performance degrades, the same happens with whole cluster.

- Heterogeneous-database replication. It is not here yet, and I do not know what’s in Codership roadmap, but group manager protocol in Galera is database independent, and it’s only matter of database drivers. For InnoDB currently it is set of patches, and I see it is quite possible to make the same for Postgres. So MySQL-Postgres cluster setup is not so far ahead

On “Company page” Codership says their goal is “to promote and exploit the latest developments in computer science to produce fast and scalable synchronous replication solution that “just works” for databases and similar applications”, which I think they have success in. Implementing fast, scalable and working group communication and transaction manager is the art.

As for now I would not put 0.7 release into production yet, but you may seriously consider to play with it in test environment, and report bugs to Codership team, they are very responsive.
I am waiting for next releases and looking to make integration with XtraDB.

Entry posted by Vadim | 6 comments

Add to: | | | |

We created a public AMI (Amazon Machine Image) with XtraDB release 8 installed on CentOS 5.3.

How to use it.

First make sure it is avaiable.

Run it. It is built for x86_64 plaform, so allowed types are m1.large, m1.xlarge and c1.xlarge

Wait till the instance starts

Now it is up and ready.

Entry posted by Aleksandr Kuzminsky | 6 comments

Add to: | | | |

At first, <tuning peak performance>. The next, <tuning purge operation> to stabilize performance  and to avoid decreasing performance.

<test condition>

Server:
PowerEdge R900, Four Quad Core E7320 Xeon, 2.13GHz, 32GB Memory, 16X2GB, 667MHz

db_STRESS:
32 sessions, RW=1, dbsize = 1000000, no thinktime

XtraDB: (mysql-5.1.39 + XtraDB-1.0.4-current)
innodb_io_capacity = 4000
innodb_support_xa = false
innodb_file_per_table = true
innodb_buffer_pool_size = 16G
innodb_read_io_threads = 8
innodb_write_io_threads = 8
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 128M
innodb_log_file_size = 512M
innodb_log_files_in_group = 2
innodb_max_dirty_pages_pct = 90
innodb_flush_method = O_DIRECT
(the followings are XtraDB specific general settings)
innodb_ibuf_active_contract = 1
innodb_adaptive_flushing = false
innodb_adaptive_checkpoint = estimate

<tuning peak performance>

At first, tuning the peak performance to use CPU and IO resource more effectively. To avoid mutex/lock contentions are good to use more CPU resource of many CPUs.

This graph shows the peak performance in tps of db_STRESS.

At current settings, "base" in the graph is the perfomance. We can confirm the mutex/lock contention roughly by the SEMAPHORES sction of SHOW INNODB STATUS output.

"xx-lock on RW-latch at 0x7f2ff40a3dc0 created in file dict/dict0dict.c line 1627"

It is index->lock, viewing the source file (and it may be HISTORY table). This is the lock for each index tree. We may be able to disperse the lock using by the partitioning of MySQL. Added the following clause to the HISTORY table definition.

"PARTITION BY HASH(REF_OBJECT) PARTITIONS 16"

Now the performance became to "+partitioned" in the graph. Looking the SEMAPHORES section again,

"has waited at handler/ha_innodb.cc line 7275 for 0.0000 seconds the semaphore:
X-lock on RW-latch at 0xd30320 created in file dict/dict0dict.c line 623"

may be the line which appears for the most times (it is dict_operation_lock). It may be partition specific lock contention. The current XtraDB has the variable to tune the contention.

innodb_stats_update_need_lock = 0 (default 1)

It skip the updating statistics which needs the lock. (it only affects for "Data_free:" value of TABLE STATS). And the performance became "+skip_stats" in the graph.  Then, the next contention at SEMAPHORES section is...

"Mutex at 0x1b3e3e78 created file trx/trx0rseg.c line 167"

may be remarkable (it is rseg->mutex). The mutex is for each rollback segments, so we can increase the rsegs to solve the contention problem. XtraDB can increase the rseg.

innodb_extra_rsegments = 64 (affects to initialization of InnoDB)

Recreated database files with the parameter. Then the performance became "+rsegs64". At last, the next contention may be "Mutex at 0x28ce8e0 created file srv/srv0srv.c line 982". It is kernel_mutex, currently we don't have proper solution for that. The setting seems to be enough for now.

<tuning purge operation>

Next, looking the sequential result in more long term.

The next problem is "History list length" growing to huge size. The value is the number of entries in rollback segment. The entries are used for consistent reading of the older transactions. They can be removed when any transactions doesn't refer the entry. This removing operation for the entries is called "purge" in InnoDB. The purge operation should be done enough on time, because the huge history list affects to performance.

Basically, the purging is done by master_thread (general background thread of InnoDB). The huge history list makes the purge operation slow, and it interferes  with the other tasks of the master_thread (e.g. flushing dirty blocks, treating insert buffer, etc...). Dimitri implemented a purge_thread to devote to the purging, and also XtraDB has similar purge_thread. Though it seems to make the throughput stabilize, it is not enough still for heavy update workloads. A single purge_thread on one CPU is not enough for updates from user threads on the all of other CPUs.

XtraDB can increase the purge_threads from the next release.

innodb_use_purge_thread = 4

seems to be enough for this workload on the server.

The first graph of followings is sequential throughput [tps] up to 3500 sec.

The next is tracking the "History list length" at the same time.

"Norm 1.0.4": Normal InnoDB Plugin 1.0.4 without XtraDB specific options
"xtra p_t 0": XtraDB 1.0.4-new (no purge_thread)
"xtra p_t 1": XtraDB 1.0.4-new (single purge_thread similar to Dimitri's)
"xtra p_t 4": XtraDB 1.0.4-new (4 purge threads)

The graphs show...

• The purge thread (> 0) helps to stabilize the throughput greatly.
• Increasing the purge threads can suppress the strong growing of the hitory list
• The adaptive checkpoint "estimate" needs the purge_thread... (than the adaptive_flushing does)

And the last 300secs' average tps are...

"Norm 1.0.4": 5725.47
"xtra p_t 0": 4699.33
"xtra p_t 1": 7130.3
"xtra p_t 4": 9118    (about 60%up from Normal Plugin 1.0.4)

In the end, the faster and more stable performance of db_STRESS benchmark is obtained by these tunings of XtraDB.

-----------------------------------------

(Added 2009.10.29)

<FAQ: Is XtraDB slower than Plugin?>

I'd like to say "no" to this question. We have been adding many tuning options to XtraDB. But they are effective not for all cases, sometimes the performance may get worse because of "not proper" or "too much value". We should choose the options correctly. XtraDB is based on InnoDB Plugin and we can set XtraDB same to InnoDB Plugin at least. The following graphs are results of XtraDB and Plugin with same options and same database.

top-left (same condition to above graphs):
innodb_flush_log_at_trx_commit = 2
innodb_doublewrite = true

top-right:
innodb_flush_log_at_trx_commit = 1
innodb_doublewrite = true

bottom-left:
innodb_flush_log_at_trx_commit = 2
innodb_doublewrite = false

bottom-right:
innodb_flush_log_at_trx_commit = 1
innodb_doublewrite = false

It seems that XtraDB is not slower than Plugin here at least.

We can start tuning based on these performances using XtraDB specific options!

Why do you make XtraDB slower than Plugin?

Entry posted by Yasufumi | 9 comments

Add to: | | | |

First is Yves Trudeau, about whom I can say many things:

• One of the top MySQL Cluster (NDB Cluster) experts in the world.
• An expert on all things High Availability, including DRBD and Heartbeat.
• Many years of experience with Huge Data.
• Half of the Waffle Grid team.
• A really nice person!

Yves joins us after a tenure of several years as a senior consultant at Sun/MySQL. Together with Matt Yonkovit, he plans to work on WaffleGrid (but as a new project under a new name, to be determined), and integration with XtraDB. Yves lives in Quebec with his family.

Next is Fernando Ipar. Fernando is our first dedicated Shift Support Captain^[1]. Fernando specializes in MySQL, GNU/Linux, systems administration, and high availability. Fernando has been involved in computer programming since he was 11, and in Open Source since 2001. He loves shade-grown coffee, music, and natural history. He lives in Uruguay with his wife and family. Fernando brightens our IRC channel with his helpful and cheerful presence!

Welcome, Yves and Fernando!

[1] Shift Support Captain is kind of a dispatcher for 24-hour service. We have been doing 24x7 service through an on-call phone number for years, but a few months ago we started literally manning the desk 24x7 -- something that we have grown large enough to both need and justify, at our current size of around 30 people. Our customers can now use email, Skype, AOL messenger, or Google Chat (as well as the phone number) to get immediate attention, 24x7, from a technically savvy person.

Entry posted by Baron Schwartz | 3 comments

Add to: | | | |

In the case of heavy updating workload, the modified ages of the many blocks are clustered. And to reduce the maximum of the modified ages InnoDB needs to flush many of the blocks in a short time, if these are not flushed at all. Then the flushing storm affect the performance seriously.

We suggested the "adaptive_checkpoint" option of constant flushing to avoid such a flushing storm. And finally, the newest InnoDB Plugin 1.0.4 has the new similar option "adaptive_flushing" as native.

Let's check the adaptive flushing options at this post.

HOW THEY WORK

< adaptive_checkpoint=reflex (older method)>

< adaptive_checkpoint=estimate (newer method)>

* estimate blocks to flush based on...

• how many modified and unflushed blocks
• progress speed of the transaction log
• the modified age average of all blocks to flush

< adaptive_flushing (default "true" at 1.0.4)>
Its behavior is not based on the oldest modified age %.

* the how many blocks to flush based on...

• how many modified and unflushed blocks
• progress speed of the transaction log
• transaction log capacity

(and adjust along with another flushing)

And it doesn't exceed the equivalent to "constant flushing (strong)"
of the "adaptive_checkpoint=reflex"

RESULTS

TPC-C like workload (100WH: 16 session full)

innodb_buffer_pool_size = 16G
innodb_max_dirty_pages_pct = 90

innodb_log_file_size = 512M
innodb_log_files_in_group = 2

innodb_io_capacity = 4000
innodb_read_io_threads = 8
innodb_write_io_threads = 8
innodb_flush_method = O_DIRECT

innodb_thread_concurrency = 0
innodb_ibuf_active_contract = 1

<none>
innodb_adaptive_flushing = false
innodb_adaptive_checkpoint = none

<flushing>
innodb_adaptive_flushing = true

<reflex>
innodb_adaptive_flushing = false
innodb_adaptive_checkpoint = reflex

<estimate>
innodb_adaptive_flushing = false
innodb_adaptive_checkpoint = estimate

case1: "innodb_doublewrite = false"

[0~1800 sec.]

[1200~1800 sec.]

averages (1200~1800 sec.)
none:     6868.92
flushing: 6655.92
reflex:     6481
estimate: 6575.88

case2: "innodb_doublewrite = true"

[0~1800 sec.]

[1200~1800 sec.]

averages (1200~1800 sec.)
none:     6569.48
flushing: 5090.12
reflex:     6871.9
estimate: 6609.9

CONSIDERINGS

The new "adaptive_flushing" seems to be tuned for "innodb_doublewrite = false" only.
(cause too much flushing for "innodb_doublewrite = true")

"innodb_adaptive_checkpoint = reflex" and "adaptive_flushing" need tuning innodb_io_capacity properly.
(The result is based on proper value "innodb_io_capacity = 4000")

"innodb_adaptive_checkpoint = estimate" is not depend on innodb_io_capacity, and it seems more "soft" result than the other methods.

So,

The "adaptive_flushing" seems good when we use "innodb_doublewrite = false".

"innodb_adaptive_checkpoint = estimate" may be safe for "innodb_doublewrite = true" for now.

I should adjust "adaptive_flushing" for "innodb_doublewrite = true".

I have tried to improve the performance. But I couldn't adjust performance of adaptive_flushing with doublewrite... So, instead of that, estimate is adjusted more.

case1: "innodb_doublewrite = false"

[0~1800 sec.]

case2: "innodb_doublewrite = true"

[0~1800 sec.]

The new "innodb_adaptive_checkpoint = estimate" seems to be more stable than before. The next question is, "Why is estimate + doublewrite better than the other no doublewrite settings?"...

Also tested in another configuration to check the behaviors in another balance of IO bound.

<changes>
innodb_buffer_pool_size = 2G   (1/8)
innodb_log_file_size = 128M    (1/4)

case1: "innodb_doublewrite = false"

[0~1200 sec.]

case2: "innodb_doublewrite = true"

[0~1200 sec.]

"innodb_adaptive_checkpoint = estimate" seems stable but seems to flush much in its initial phase. The adaptive_flushing seems to be more soft in this cases.

Hmmm....

We should choose suitable adaptive_xxx for each workload for now...

Entry posted by Yasufumi | 12 comments

Add to: | | | |

Last week I talked about why you don't want to shard. This week I'm following up with the top 10 enhancements that XtraDB has over the built-in InnoDB included in MySQL 5.0 and 5.1.  Building this list was not really a scientific process - It's always difficult to say which feature is better than another, because a lot of it depends on the individual workload.  My ranking method was to pick the features that have the highest impact and are most applicable to all workloads first:

1. CPU scalability fixes - XtraDB improves performance on systems with multi-cpus (see docs 1, 2).
2. Import/Export Tables - XtraDB allows you to import an arbitrary  table from one server to another, by backing up the .ibd file with Xtrabackup (see docs).
3. IO scalability fixes - A lot of the internal algorithms of InnoDB are based on the non-configurable assumption that the server has only a single disk installed (100 iops).  One problem in particular that this causes, is that InnoDB the algorithm which chooses if InnoDB is too busy to flush dirty pages can consider it's self busy very easily.  Keeping a large percentage of pages dirty increases recovery time, and will lead to more work when checkpoints are eventually forced at the end of a log file.  XtraDB improves this with innodb_io_capacity, as well as configuration items for innodb_read_threads, innodb_write_threads (see docs).
4. Better Diagnostics - The SHOW ENGINE INNODB STATUS command in XtraDB shows a lot more information than the standard InnoDB status (see docs).  The built-in InnoDB status also has some problems with the placement of items (a long transaction list will prevent the rest of the information shown).  In addition to this, XtraDB diagnostics include the ability to see the contents of the buffer pool (see docs), and InnoDB row statistics are inserted into the slow query log (see docs).
5. Fast Crash Recovery - In the built-in InnoDB, the crash recovery process is sometimes best measured in hours and days - this restricts users to using very small transaction log files (innodb_log_file_size), which is worse for performance.  In a simple test, XtraDB recovered ten times faster (see docs).
6. InnoDB Plugin Features - XtraDB is derived from the InnoDB plugin, which has fast index creation (as opposed to recreating the whole table!) and page compression.
7. Adaptive Checkpointing - The built-in InnoDB can have erratic dips in performance as it approaches the end of a log file and needs to checkpoint - which can cause a denial of service to your application (this can be seen in any benchmark - such as this one).  In XtraDB, adaptive checkpointing can smooth out the load, and checkpoint data more aggressively as you approach the end of a log file (see docs).
8. Insert Buffer control - The insert buffer is a great feature of InnoDB that is not often discussed.  It allows you to delay the writing of non-unique secondary index pages, which can often lead to a lot of merged requests and reduced IO.  The  problem with the insert buffer in the built-in InnoDB, is that there are no options to tweak it.  It can grow to 1/2 the size of your buffer pool, and when it does, it doesn't try to aggressively free entries (a full buffer provides no use) or reduce its size (see docs).
9. Data dictionary control - Once an InnoDB table is opened it is never freed from the in-memory data dictionary (which is unlimited in size).  XtraDB introduces a patch to be able to control this, which is useful in cases where users have a large number of tables. (see docs).
10. Additional undo slots - In the built-in InnoDB, the number of open transactions is limited to 1023 (see bug report).  XtraDB allows this to be expanded to 4072 (Warning: Incompatible change!).  (see docs).

All of these 10 items will be covered in our Training workshops for InnoDB and XtraDB.  In Santa Clara / San Francisco between 14-16 September?  Come along!

My next post in this series will be on XtraDB: The Top 10 Configuration Parameters.

Entry posted by Morgan Tocker | 5 comments

Add to: | | | |

Entry posted by Vadim | One comment

Add to: | | | |

Now I am going to show how to use it (the video will be also available on percona.tv).
Let's take tpcc schema and running standard MySQL ® 5.0.83, and assume we want to copy order_line table to different server. Note I am going to do it online, no needs to lock or shutdown server.

To export table you need XtraBackup, and you can just specify table or table by mask:

Now in /data/vadim/mysql/export we have backup but only with ibdata1 and order_line.ibd files

Second step is to prepare backup, but with special option "export"

When it's done we have two files order_line.ibd and order_line.exp in directory /data/vadim/mysql/export/tpcc. ibd is regular InnoDB ® file, and exp is file with special export information. Both files should be copied to remote server.

Now on remote server you have to run MySQL server ® with XtraDB6 storage engine, you can take one of our binary builds with 5.1.36.

On new server we run
set global innodb_expand_import=1; to put XtraDB into extended import mode, and now we need to create empty table with the same table definition as on old servers:

Next step is to remove created .ibd file, it's done by command:

And now you copy both order_line.ibd and order_line.exp to MysqlDataDir/testimport dir. When it's done, final import command:

now some magic happens and you can see progress of import in error.log

When finished quick check

New table has been imported!

Entry posted by Vadim | 3 comments

Add to: | | | |

Today we are pleased to announce release 6 of XtraDB - the result of 2 months hard work.

The release includes following new features:

• MySQL 5.1.36 as a base release
• New patch innodb_recovery_patches.patch
• Experimental adaptive checkpoint method estimate
• innodb_stats - the implementation of the fix forMySQL Bug#30423
• expand-import Support of import InnoDB / XtraDB tables from another server
• split-bufpool-mutex-3 New patch to split buffer pool mutex
• g-style-io-thread Google's fixes to InnoDB IO threads
• dict-size-limit Limit of internal data dictionary

Fixed bugs:

• MySQL Bugs: #39793: Foreign keys not constructed when column has a '#' in a comment or default value
• Bug #395784 in Percona-XtraDB: “main.innodb_xtradb_bug317074 internal check fails on 5.1.36”
• Bug #395778 in Percona-XtraDB: “main.innodb-analyze internal check failed on 5.1.36”
• Bug #391189 in Percona-XtraDB: “main.innodb_bug36172 mysq test fails with innodb_stat.patch applied”
• Bug #388884 in Percona-XtraDB: “patching fails on 5.1.35”

The builds for RedHat4,5 and Debian are located on http://www.percona.com/mysql/xtradb/5.1.36-6/
The latest source code of XtraDB, including development branch you can find on LaunchPAD.

Please report any bugs found on Bugs in Percona XtraDB Storage Engine for MySQL.
For general questions use our Pecona-discussions group, and for development question Percona-dev group.

For support, commercial and sponsorship inquiries contact Percona

Entry posted by Aleksandr Kuzminsky | 6 comments

Add to: | | | |