At first, <tuning peak performance>. The next, <tuning purge operation> to stabilize performance  and to avoid decreasing performance.

<test condition>

Server:
PowerEdge R900, Four Quad Core E7320 Xeon, 2.13GHz, 32GB Memory, 16X2GB, 667MHz

db_STRESS:
32 sessions, RW=1, dbsize = 1000000, no thinktime

XtraDB: (mysql-5.1.39 + XtraDB-1.0.4-current)
innodb_io_capacity = 4000
innodb_support_xa = false
innodb_file_per_table = true
innodb_buffer_pool_size = 16G
innodb_read_io_threads = 8
innodb_write_io_threads = 8
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 128M
innodb_log_file_size = 512M
innodb_log_files_in_group = 2
innodb_max_dirty_pages_pct = 90
innodb_flush_method = O_DIRECT
(the followings are XtraDB specific general settings)
innodb_ibuf_active_contract = 1
innodb_adaptive_flushing = false
innodb_adaptive_checkpoint = estimate

<tuning peak performance>

At first, tuning the peak performance to use CPU and IO resource more effectively. To avoid mutex/lock contentions are good to use more CPU resource of many CPUs.

This graph shows the peak performance in tps of db_STRESS.

At current settings, “base” in the graph is the perfomance. We can confirm the mutex/lock contention roughly by the SEMAPHORES sction of SHOW INNODB STATUS output.

“xx-lock on RW-latch at 0×7f2ff40a3dc0 created in file dict/dict0dict.c line 1627″

It is index->lock, viewing the source file (and it may be HISTORY table). This is the lock for each index tree. We may be able to disperse the lock using by the partitioning of MySQL. Added the following clause to the HISTORY table definition.

“PARTITION BY HASH(REF_OBJECT) PARTITIONS 16″

Now the performance became to “+partitioned” in the graph. Looking the SEMAPHORES section again,

“has waited at handler/ha_innodb.cc line 7275 for 0.0000 seconds the semaphore:
X-lock on RW-latch at 0xd30320 created in file dict/dict0dict.c line 623″

may be the line which appears for the most times (it is dict_operation_lock). It may be partition specific lock contention. The current XtraDB has the variable to tune the contention.

innodb_stats_update_need_lock = 0 (default 1)

It skip the updating statistics which needs the lock. (it only affects for “Data_free:” value of TABLE STATS). And the performance became “+skip_stats” in the graph.  Then, the next contention at SEMAPHORES section is…

“Mutex at 0×1b3e3e78 created file trx/trx0rseg.c line 167″

may be remarkable (it is rseg->mutex). The mutex is for each rollback segments, so we can increase the rsegs to solve the contention problem. XtraDB can increase the rseg.

innodb_extra_rsegments = 64 (affects to initialization of InnoDB)

Recreated database files with the parameter. Then the performance became “+rsegs64″. At last, the next contention may be “Mutex at 0×28ce8e0 created file srv/srv0srv.c line 982″. It is kernel_mutex, currently we don’t have proper solution for that. The setting seems to be enough for now.

<tuning purge operation>

Next, looking the sequential result in more long term.

The next problem is “History list length” growing to huge size. The value is the number of entries in rollback segment. The entries are used for consistent reading of the older transactions. They can be removed when any transactions doesn’t refer the entry. This removing operation for the entries is called “purge” in InnoDB. The purge operation should be done enough on time, because the huge history list affects to performance.

Basically, the purging is done by master_thread (general background thread of InnoDB). The huge history list makes the purge operation slow, and it interferes  with the other tasks of the master_thread (e.g. flushing dirty blocks, treating insert buffer, etc…). Dimitri implemented a purge_thread to devote to the purging, and also XtraDB has similar purge_thread. Though it seems to make the throughput stabilize, it is not enough still for heavy update workloads. A single purge_thread on one CPU is not enough for updates from user threads on the all of other CPUs.

XtraDB can increase the purge_threads from the next release.

innodb_use_purge_thread = 4

seems to be enough for this workload on the server.

The first graph of followings is sequential throughput [tps] up to 3500 sec.

The next is tracking the “History list length” at the same time.

"Norm 1.0.4": Normal InnoDB Plugin 1.0.4 without XtraDB specific options
"xtra p_t 0": XtraDB 1.0.4-new (no purge_thread)
"xtra p_t 1": XtraDB 1.0.4-new (single purge_thread similar to Dimitri's)
"xtra p_t 4": XtraDB 1.0.4-new (4 purge threads)

The graphs show…

• The purge thread (> 0) helps to stabilize the throughput greatly.
• Increasing the purge threads can suppress the strong growing of the hitory list
• The adaptive checkpoint “estimate” needs the purge_thread… (than the adaptive_flushing does)

And the last 300secs’ average tps are…

"Norm 1.0.4": 5725.47
"xtra p_t 0": 4699.33
"xtra p_t 1": 7130.3
"xtra p_t 4": 9118    (about 60%up from Normal Plugin 1.0.4)

In the end, the faster and more stable performance of db_STRESS benchmark is obtained by these tunings of XtraDB.

—————————————–

(Added 2009.10.29)

<FAQ: Is XtraDB slower than Plugin?>

I’d like to say “no” to this question. We have been adding many tuning options to XtraDB. But they are effective not for all cases, sometimes the performance may get worse because of “not proper” or “too much value”. We should choose the options correctly. XtraDB is based on InnoDB Plugin and we can set XtraDB same to InnoDB Plugin at least. The following graphs are results of XtraDB and Plugin with same options and same database.

top-left (same condition to above graphs):
innodb_flush_log_at_trx_commit = 2
innodb_doublewrite = true

top-right:
innodb_flush_log_at_trx_commit = 1
innodb_doublewrite = true

bottom-left:
innodb_flush_log_at_trx_commit = 2
innodb_doublewrite = false

bottom-right:
innodb_flush_log_at_trx_commit = 1
innodb_doublewrite = false

It seems that XtraDB is not slower than Plugin here at least.

We can start tuning based on these performances using XtraDB specific options!

Why do you make XtraDB slower than Plugin?

Entry posted by Yasufumi | 9 comments

Add to: | | | |

Beside ideas I put before http://www.mysqlperformanceblog.com/2009/03/30/my-hot-list-for-next-innodb-features/ (and one of them – moving InnoDB tables between servers are currently under development), we have few mores:

- Stick some InnoDB tables / indexes in buffer pool, or set priority for InnoDB tables. That means tables with bigger priority will be have more chances to stay in buffer pool then tables with lower priority. Link to blueprint https://blueprints.launchpad.net/percona-patches/+spec/lru-priority-patch

- Separate LRU list into several lists, and in this way it will allow us to emulate several buffer pool, with features to keep different tables in different buffer pools and also to decrease contention on buffer pool. Link https://blueprints.launchpad.net/percona-patches/+spec/multiple-lru-patch

- We are looking to include Waffle Grid into XtraDB releases with some additional features like caching buffer pool on SSD.

If ideas are interesting for you and you want to support them, contact us

Entry posted by Vadim | 7 comments

Add to: | | | |

For me it was a big wall in using these patches that they were part of one big patch, which you can apply only to 5.0.37, and now there is no barrier to include patches into our builds or MySQL releases.

If you do not know what is Global Transactional ID is – it is worth to look http://code.google.com/p/google-mysql-tools/wiki/GlobalTransactionIds. From my point of view – it is absolutely new view on MySQL replication and it can change MySQL replication architecture.

We definitely will look if we can integrate patches into percona builds and provide binaries if there is no problems. Also I mostly sure the patches will be included into MariaDB along with other Percona-improvements and XtraDB storage engine.

Entry posted by Vadim | 9 comments

Add to: | | | |

Entry posted by Morgan Tocker | 4 comments

Add to: | | | |

Our idea is not new, it was taken from TransactionalReplication http://code.google.com/p/google-mysql-tools/wiki/TransactionalReplication patch and http://bugs.mysql.com/bug.php?id=34058 bug report. Basically we want to store the state of replication in InnoDB transactional log file, in this case we will be able to see what position in replication the last executed transaction corresponds to. Of course it will work if you have writes exclusively to InnoDB storage engine, case with mix of storage engines is much more complex and I do not see easy way to solve it.

So we propose overwrite_relay_log_info extension for XtraDB storage engine, the name comes from fact that XtraDB will try to rewrite relay-log.info by replication position, or at least this info is available in error-log output, so you can repoint your slave to correct position by executing CHANGE MASTER command.

Currently patch is available in Launchpad lp:~percona-dev/percona-xtradb/overwrite-relay-log-info , and after testing it will go to main XtraDB tree. If there is interest the patch will be ported to 5.0 and 5.1 trees. Some info available there http://www.percona.com/docs/wiki/percona-xtradb:innodb_overwrite_relay_log_info

Entry posted by Vadim | 13 comments

Add to: | | | |

Patch was made by request of our customer Vertical Response and released under GPL, so you can download it there http://mysqlperformanceblog.com/files/patches/innodb_dict_size_limit_standalone.patch. Currently patch is on testing stage, but later will be included into our releases. To limit memory we introduce new variable innodb_dict_size_limit (in bytes).

Some internals: There is already implemented in InnoDB LRU-based algorithm to keep only recent table entries, but it was not used by reason that InnoDB has to know if table is used or not on MySQL level. We made it by checking MySQL table_cache. If table is placed in table_cache we consider it as used, if not – we can delete it from InnoDB data dictionary. So there is the trick – if you have big enough table_cache, memory consumed by data dictionary may exceed innodb_dict_size_limit, as we can’t delete any table entry from it.

To finalize this post small marketing message – if you faced bug or problem which exists for long time and is not going to be solved by MySQL / InnoDB – contact us regarding Custom MySQL Development.

Entry posted by Vadim | 13 comments

Add to: | | | |

The problem is the InnoDB tries to keep the same order of transactions in binary logs and in transaction logs and acquires mutex to serialize writes to both logs. We basically propose to break this serialization – in XtraDB release3 (will be announced soon, you can take current version for testing from Launchpad) we introduce –innodb-unsafe-group-commit mode. There are results with this options vs without (results are in transactions per second, more is better, this is sysbench OLTP load).

I tested it on Dell PowerEdge R900 with RAID 10 in WriteThrough mode to emulate absence of BBU. With BBU you will not see this problem (all results will scale well) as internal RAID cache will accumulate changes and return fsync() call immediately without real syncing data in disk.

So what can be wrong if you run –innodb-unsafe-group-commit — as I said there is possibility that transactions in binary-logs will be in different order than in InnoDB transactional log. Why this is bad? For example if box crashes and InnoDB does recovery: transactions on slaves may be executed in different order — that is you MAY get slaves unsynchronized with master. Is performance benefit worth it? It’s up to you, but I think better to have this choice then do not have.

I do not urge to use –innodb-unsafe-group-commit, I propose to have BBU on your RAID. But if it appears you don’t have it, and write load on server is significant — it may worth to try –innodb-unsafe-group-commit.

Entry posted by Vadim | 11 comments

Add to: | | | |

against table

with table size 1mil records fully fitting in memory. I run it with innodb_thread_concurrency=16 to match threads inside InnoDB with count of cores.

Results for InnoDB-plugin-1.0.2 were kind of discouraging, dropping down just after 8 connections, si I decided to test MySQL-5.1.30 with standard InnoDB. It was better, but still far from what we can expect.

After investigation Yasufumi pointed to page_hash mutex, which was abused. This mutex is used as mutex even in places where shared read lock is enough - and we replaced page_hash mutex to page_hash read-write lock.

The new results you can see on graph.

The patch for page_hash will be included in next release of XtraDB, and expect new results.

for reference InnoDB was run with next parameters:

Entry posted by Vadim | 13 comments

Add to: | | | |

Fixes include:

Control of InnoDB insert buffer to address problems Peter mentioned http://www.mysqlperformanceblog.com/2009/01/13/some-little-known-facts-about-innodb-insert-buffer/, also check Bug 41811 to see symptoms of problem with Insert buffer.

http://www.percona.com/docs/wiki/patches:innodb_io_patches

* innodb_flush_neighbor_pages (default 1) - When the dirty page are flushed (written to datafile), this parameter determines whether the neighbor pages in the datafile are also flushed at the same time or not. If you use the storage which don't have “head seek delay” (e.g. SSD or enough Write-Buffered), 0 may show better performance. 0:disable, 1:enable

* innodb_ibuf_max_size (default [the half of innodb_buffer_pool_size](bytes)) - This parameter is startup parameter. If the lower value is set than the half of innodb_buffer_pool_size, it is used as maximum size of insert buffer. To restrict to the too small value (e.g. 0) is not recommended for performance. If you don't like the insert buffer growing bigger, you should use the following parameters instead. (* If you use very fast storage, small value (like several MB) may show better performance.)

* innodb_ibuf_accel_rate (default 100(%)) - This parameter is additional tuning the amount of insert buffer processing by background thread. Sometimes, only innodb_io_capacity is insufficient to tune the insert buffer.

* innodb_ibuf_active_contract (default 0) - By default (same to normal InnoDB), the each user threads do nothing about contracting the insert buffer until the insert buffer reaches its maximum size. 1 makes the each user threads positive to contract the insert buffer as possible in asynchronous.

Second important fix introduces variable use_global_long_query_time, which allows all current threads see change of long_query_time. By default value set in SET GLOBAL long_query_time=N command is visible only on new established connection, which is problem if you have pre-established connection pool, say in Java or Ruby on Rails application. With use_global_long_query_time=true even all current threads will respect SET GLOBAL long_query_time=N. The feature made for EngineYard, hosting provider for Ruby on Rails application.

You can download binaries (RPMS x86_64) and sources with patches here
http://www.percona.com/mysql/5.0.75-b12/

Entry posted by Vadim | 17 comments

Add to: | | | |

We're involved in a dozen or so open-source projects, but our three core efforts at the moment are the following:

• Percona patches, which are included in our own MySQL builds and then in OurDelta builds and perhaps others as well
• XtraDB, which is our new high-performance transactional storage engine
• Maatkit, which is a toolkit that provides advanced functionality for MySQL.

We have a team of dedicated MySQL developers working on the server and on ExtraDB, and a dedicated Maatkit developer. Other Percona employees also put significant time into these projects.

Outside observers have commented that our development process doesn't seem very open-source.  That is, we typically just go build the software and then announce it. We don't involve the community very much in our decisions about what features to include, or how they should get built; and we don't encourage community contributions directly into our codebase.  There's also some ambiguity about where the money comes from and where it goes.  These are all fair points to bring up.  Peter, Vadim and I thought we should address them and let everyone know how we really work on these things and what our vision for the future is.

How We Decide On Features

There are at least three sources of input into how we prioritize our work.

1. Clear and present needs. If we are crippled day in and day out by the lack of some feature, or the presence of some bug, then we will take the initiative to fix it. The most obvious example is our enhanced slow-logging functionality, which is part of our standard patches. These patches make it possible to learn a great deal about what your server is doing and how to optimize it.
2. Customer demands. Customers come to us and ask for the features that they see as clear and present needs; or they come to us with particular performance issues and we analyze it and find that the best solution is to modify the server. Many of the other features in our patch set are in this category, as is a lot of ongoing work that's not released yet.
3. The general public. People comment on our blog posts, or send us email, and so forth. We notice these requests, and if they come up often enough and we think they'll be useful, they start to stick in our brains and we eventually put them on the TODO list.

The order of priorities is this: paying customers, obvious needs for our consulting practice, and the public wishlist. However, even when work is sponsored by paying customers, we don't just blindly put the features into our Percona builds. Firstly, customers get to choose whether they want us to do this. Secondly, the builds are designed to meet real-world needs and we are working very carefully to enhance the MySQL server but not end up with a chaotic mix of random stuff.

Some features are also so specific to certain customers that we don't include them. When we do target an improvement for our builds, it's a big win for everyone, because we share the development cost with the customer (you can think of it as an open-source discount; it is us contributing towards the public good) and we maintain the feature in subsequent minor versions of the server free of charge. Features that we don't include in our public patch set don't get forward-ported for free, so customers pay if they want a version that is ported to a newer server.

In a separate category, there's also work that we do for customers which is not open-source. We are currently customizing the MySQL server and storage engines for some of Silicon Valley's most innovative startups. I won't talk about this any more, but I wanted to mention it for the sake of full disclosure.

How We Support Our Work

This work is usually supported by our customers. (It happens, but not often, that someone just writes new functionality for free in spare time.) However, if the work is open-source, we generally run this as a "less-for-profit" part of our business. I'll clarify that in a minute.

There are generally two ways that we channel money to features. The first is when a customer says "I want this" and pays us to build it. In these cases, we work on a schedule to deliver the required features. We treat it as developer-for-hire with all the normal expectations of being hired. If we all agree that the result will be open-sourced and the general public will benefit, then we work for a discount. We're still working for a profit -- part of our business is custom MySQL development, after all -- but we discount it.

The second way is when people say "here's some money for the general fund to support your future planned features." In this case, the money goes into the queue for when someone is available to work on the modifications, and we either pay attention to any preferences the donor has, or we use our judgment as to which enhancements will add the most value to our builds. In practice, this second model has not happened very often; we have used it more for Maatkit and MMM than for server patches. But we hope that will change as more people find out about the possibility for getting their wishes. This model lets us work on features that everyone wants but no one wants to pay for. And in this case we work on a non-profit basis: the developer who works on the features is paid at a special rate for "internal work" (as opposed to working for "external customers").

You can also think about this as dual sources for the money to support the open-source efforts. One is customers who channel money directly to development. (Thanks!) The other is all of our customers who are indirectly sponsoring the development just by being customers. In other words, if you're a customer, you're contributing to open-source software development. (Thanks to you, too!)

If you would like Percona to create custom MySQL features or builds for you, please contact us.

Why We Don't Grant Commit Rights

As far as we know, no one has ever asked for commit rights to our codebase.

That aside, the Percona contributions to open-source are not a potluck dinner where everyone brings something and puts it on the table. These builds are stamped with our name and are downloadable from our website. They're a core part of our offering to customers (and non-customers can get them for free, too). If you're a customer, we stand behind the builds and we support them. The builds power some of the most demanding MySQL installations in the world. All code has to be vetted very carefully, and all features have to deliver a compelling value.

There's an overarching reason, though. Percona is not the ultimate upstream source. Community contributions ought to be going into the MySQL codebase, not the Percona codebase. Percona's relationship to the rest of the community is that of a peer, not a parent.

Perhaps some of you are wondering, "why don't you contribute your patches back to MySQL?"

We do contribute our patches. They are GPL-licensed, which is both generous and fair, and MySQL is free to take them without even asking permission, as long as they abide by the GPL. We welcome everyone to use our patches. OurDelta does so -- it is a great example of how the community can aggregate patches and make something greater than the sum of the parts.

When it comes to Sun/MySQL, we actively and openly talk to them about it, and it's an ongoing conversation. However, there are various reasons why it's not the ultimate solution for our customers, and their interests come first. For one thing, getting patches accepted is a somewhat laborious process, and we don't have the resources for it ourselves. We could do it, but we'd have to pass the costs on to our customers.

The second thing is that it takes far too long for patches to actually show up in a GA version of the server. Customers pay us to solve their problems now. This means that one way or another, we're going to be maintaining the custom patches in our own tree and making our own builds, and there is no sense in doing a halfway job of it. So that removes the motivation for us or our customers to try to get Sun/MySQL to accept the patches -- it wouldn't add any extra value. We'd like Sun/MySQL to accept the patches, but with the customers' needs solved, the onus is on them to do it, which seems fair -- after all, Percona and Percona customers already co-sponsored the development, and Sun/MySQL will be reselling the results.

This is a large and contentious topic that has been discussed at great length elsewhere and is not appropriate to get into here in its full scope. But I'll point out that some of our patches have been accepted in the past and there are signs that this might continue. We certainly hope so.

How You Can Get Your Wishes To Come True

If you have a particular need for some modification to the MySQL server, XtraDB, Maatkit, our InnoDB data recovery tools, MMM, innotop, or even projects like standard GNU utilities, we're listening. You can contact us through our website, and offer to sponsor the work. There are also mailing lists for our various projects.

Here's an example: wouldn't it be great if there were a built-in type in MySQL that could store date and time with microsecond precision? Lots of people have asked for that. If you all put some money in the hat, it could become a reality.

Where It All Begins

Our open-source efforts flow from our company values. We believe that software and information wants to be free. We also support innovation and entrepreneurship. We like to combine these things when we can! The complexity of our customers' different situations leads to the need for a lot of flexibility in our approach, which is why there are so many nuances. And we generally try to create incentive towards what we see as the public good. That's the motivation behind our shared-development-cost model, for example -- it creates an incentive for customers to say "yes" when we ask them to agree to open-source the features.

Open-source is not a passing fad. It's not a marketing gimmick for us. It's who we are at the core. Funding its development in a fair way is a great balancing act, and finding a balance that seems to make sense for everyone is very rewarding.

Entry posted by Baron Schwartz | 3 comments

Add to: | | | |