MySQL Performance Blog » xtradb

Maximal write througput in MySQL

Vadim — Mon, 01 Mar 2010 06:53:08 +0000

I recently was asked what maximal amount transactions per second we can get using MySQL and XtraDB / InnoDB storage engine if we have high-end server. Good questions, though not easy to answer, as it depends on:

- durability setting ( innodb_flush_log_at_trx_commit = 0 or 1 ) ?
- do we use binary logs ( I used ROW based replication for 5.1)
- do we have sync_binlog options.

So why would not take these as variable parameters and run simple benchmark.
I took sysbench update_key scenario ( update indexed field on simple table)
and used Dell PowerEdge R900 with 16 cores, FusionIO as storage for table and RAID 10 with BBU as storage for innodb log files, innodb system table space and binary logs. And I used Percon-XtraDB-5.1.43-9.1 for benchmarks. All used partitions are formatted in XFS and mounted with nobarrier option.

I run update key for various threads and with next parameters

trx_commit=0 : innodb_flush_log_at_trx_commit = 0 and no binary logs
trx_commit=1 : innodb_flush_log_at_trx_commit = 1 and no binary logs
trx_commit=0 & binlog : innodb_flush_log_at_trx_commit = 0 and binary logs
trx_commit=1 & binlog : innodb_flush_log_at_trx_commit = 1 and binary logs
trx_commit=1 & binlog & sync_bin : innodb_flush_log_at_trx_commit = 1 and binary logs and sync_binlog=1

There are results I get:

I found results being quite interesting.
with innodb_flush_log_at_trx_commit = 0 maximal tps is 36332.02 tps, which drops to 23115.04 tps as
we switch to innodb_flush_log_at_trx_commit = 1. As we use RAID10 with BBU, I did not expect the drops is going to be significant. In second case InnoDB spends

With enabling binary logs, the results drops to 17451.01 tps with innodb_flush_log_at_trx_commit = 0 and to 12097.39 tps with innodb_flush_log_at_trx_commit = 1. So with binary logs serialization is getting even worse.

Enabling sync_binlog makes things really bad, and maximal results I have is
3086.7 tps. So this is good decision if binary log protection is worth such drop.

UPDATE ( 3/4/2010 )

Results with innodb_flush_log_at_trx_commit = 2

Results with innodb_flush_log_at_trx_commit = 2 and binlogs

Entry posted by Vadim | 7 comments

Add to: | | | |

XtraDB feature: save / restore buffer pool

Vadim — Thu, 21 Jan 2010 00:07:20 +0000

We recently released XtraDB-9, and while we did not highlight it in announcement, the release-making feature is ability to save and restore InnoDB buffer pool.
The idea is not new and was originally developed by Jeremy Cole (sorry, I do not have the link on hands) some time ago, and now we implemented it in XtraDB.

Why would we need to save and restore content of buffer pool ?
There are several reasons.
First, it’s not rate on modern servers to have 32GB+ of RAM, with allocated InnoDB buffer_pool 26GB or more. When you do restart of server, it may take long time to populate cache with useful data before you can bring it back to serve production load. It’s not rare to see
maintenance cycle takes two or more hours, mainly because the slave need to catchup with master and to warm cache.
In case with the server crash, it is even worse, you need to wait possible long time on InnoDB
recovery (we have the patch for that too, in that post you can see InnoDB recovery took 1h to accomplish) and after that warm caches.

Second, it is useful for some HA schemas, like DRBD, when, in case of failover, you need to start passive instance on cold.

So let’s see what results we have.
Details about patch you can get there http://www.percona.com/docs/wiki/percona-xtradb:patch:innodb_lru_dump_restore (Yasufumi names it LRU dump/restore, because he thinks about buffer pool as about LRU list, which how it is internally).

To save buffer pool you execute
select * from information_schema.XTRADB_ADMIN_COMMAND /*!XTRA_LRU_DUMP*/;
and to restore
select * from information_schema.XTRADB_ADMIN_COMMAND /*!XTRA_LRU_RESTORE*/;
it will create/read file ib_lru_dump from your database directory.

You may want to sort ib_lru_dump in order of pages in tablespaces, so RESTORE will be
performed in most sequential way. The small python script lrusort.py to sort ib_lru_dump is available
in our Launchpad branch lp:~percona-dev/percona-xtradb/extensions-1.0.6

I made small tpcc benchmark to show effect with restored buffer_pool (the condition of
benchmarks are the same as in my runs on fast storages, and I
used RAID10 to store InnoDB files).
First run (xtradb cold) I made just after restart and ran it for 1h.
After that I saved buffer_pool, restarted mysqld, restored buffer_pool ( it took about 4 min
to load 26GB worth of data), and run (xtradb warm) tpcc again.

Here is graphical results (results in New Transactions Per 10 sec, more is better):

As you see in the cold run it took 1500-1800 sec to enter into stable mode, while
it warm run it happened almost from start. There was some period of unstable results, but it
did not affect ability to serve load.

You are welcome to test it, it is available in XtraDB-9 release and also in MariaDB 5.1.41-RC.

Entry posted by Vadim | 15 comments

Add to: | | | |

XtraDB storage engine release 1.0.6-9

Aleksandr Kuzminsky — Thu, 14 Jan 2010 05:09:42 +0000

Dear Community,

As of today Release 9 of XtraDB storage engine is available.

The release includes following new features:

The release is base on 1.0.6 version of InnoDB plugin.
MySQL 5.1.42 as a base release
Separate purge thread and LRU dump is implemented (this feature was actually added in Release 8, but somehow it was forgotten)
New patch innodb_relax_table_creation
Added extended statistics to slow log
Adjust defaults with performance intention
Added parameter to control checkpoint age
Added recovery statistics output when crash recovery (disabled by default)
Patch to dump and restore innodb_buffer_pool

Fixed bugs:

Bug#488315:
rename columns and add unique index cause serious inconsistent with mysqld
Bug #495342: “MySQL 5.1.41 + InnoDB 1.0.6 + XtraDB patches extensions-1.0.6, rev. 127 fails to compile on OpenSolaris”
Bug #47621: change the fix of http://bugs.mysql.com/47621 more reasonable

The builds for RedHat 4, 5 , Debian and Ubuntu are located on http://www.percona.com/mysql/xtradb/5.1.42-9/ . Since Release 9 .tar.gz packages are available for FreeBSD.

There is an RPM Percona-release. It installs Percona.repo to /etc/yum.repos.d/ and the public key which we use to sign RPMs. The copy of the key is located on our web-site.

Once you install Percona-release repo the upgrade becomes as easy as

PLAIN TEXT

CODE:

yum update MySQL-server-percona MySQL-client-percona

The latest source code of XtraDB, including development branch you can find on LaunchPAD.

Please report any bugs found on Bugs in Percona XtraDB Storage Engine for MySQL.
For general questions use our Pecona-discussions group, and for development question Percona-dev group.

For support, commercial and sponsorship inquiries contact Percona

Entry posted by Aleksandr Kuzminsky | 9 comments

Add to: | | | |

InnoDB, InnoDB-plugin vs XtraDB on fast storage

Vadim — Wed, 13 Jan 2010 08:52:38 +0000

To continue fun with FusionIO cards, I wanted to check how MySQL / InnoDB performs here. For benchmark I took MySQL 5.1.42 with built-in InnoDB, InnoDB-plugin 1.0.6, and XtraDB 1.0.6-9 ( InnoDB with Percona patches).
As benchmark engine I used tpcc-mysql with 1000 warehouses ( which gives around 90GB of data + indexes) on my workhourse Dell PowerEdge R900 ( details about box ).

On storage configuration: FusionIO 160GB SLC and 320GB MLC cards are configured in software RAID0 to store InnoDB datafiles. For InnoDB logs and system tablespace I used partition on regular RAID10 with regular hard drives, here I followed Yoshinori Matsunobu's recommendations http://yoshinorimatsunobu.blogspot.com/2009/05/tables-on-ssd-redobinlogsystem.html and taking fact that FusionIO is not perfect for sequential writes http://www.mysqlperformanceblog.com/2010/01/11/fusionio-320gb-mlc-benchmarks/

Full results I put on page http://www.percona.com/docs/wiki/benchmark:fusionio:mysql:start, here are my thoughts and interesting facts.

First, chart with results for InnoDB vs InnoDB-plugin during runs (values are in new order transactions per minute, more is better) :

As you see InnoDB-plugin is doing much better here, it allows to utilize multiple IO threads,
which as we saw ( http://www.mysqlperformanceblog.com/2010/01/11/fusionio-320gb-mlc-benchmarks/ ) is necessary to get most throughput from FusionIO.

Also you may see from graph some waves for InnoDB-plugin. Here we observe innodb_adaptive_flushing in action (which is ON by default), and I think innodb_adaptive_flushing in InnoDB-plugin is not quite balanced, it may do overaggressive flushing, when it is not necessary.

But looking on CPU stats (see graph later), I guess InnoDB-plugin spends most time in buffer_pool mutex, contention here is not fully resolved yet in InnoDB-plugin.

Now, let's take XtraDB. In additional to multiple IO threads, we have patch to decrease contention on buffer_pool mutex, plus separate purge thread. Also we use different adaptive_checkpoint algorithm.

The results are:

So I guess buffer_pool improvements play here for XtraDB, and looking on summary result:

InnoDB 9439.316 NOTPM
InnoDB-plugin-1.0.6 15299.333 NOTPM
XtraDB-1.0.6-9 26160.551 NOTPM

InnoDB-plugin is 1.6x times better InnoDB, and XtraDB is 1.7x times better InnoDB-plugin.

Now on CPU usage and disk utilization.

Disk throughput:

CPU (user) usage:

Even with improvements, XtraDB performs less 150MB/s in disk writes (from benchmarks we
saw FusionIO can do much more) and with 45-50% of idle CPU.

I assume we still see significant contentions inside XtraDB, and there big room for improvements. As for InnoDB-plugin, I'd wish InnoDB team makes some actions on buffer_pool mutex problem.

Finally I wanted to check what if we put innodb transactional logs and system tablespace on FusionIO also, there is graph for that:

It is not so bad, with final result 23038.283 NOTPM, it is only about 12% worse than with logs on separate partition.

And to make reference point, I run the same but with all files on RAID10 with regular disks,
the graph is there:

with final result: 2873.783 NOTPM ( about 88% worse than all files on FusionIO)

To summarize

MySQL InnoDB/InnoDB-plugin/XtraDB is not fully able to utilize throughput of FusionIO. XtraDB is doing better job with internal contention, but much more can be done.
Still you can have very impressive performance improvement in IO-intensive or IO-bound workloads. You may want to use InnoDB-plugin or XtraDB to get better results.
Putting logs on separate partition may be good idea, but only if you have possibility to do that. Making special setup for that may be not worth improvements

Entry posted by Vadim | 10 comments

Add to: | | | |

Speaking at the LA MySQL Meetup – 18th November

Morgan Tocker — Sun, 01 Nov 2009 17:35:44 +0000

A recent photo from Highload.ru

I said in my last post, that we're interested in speaking at MySQL meetups, and I'm happy to say that the Los Angeles MySQL Meetup has taken us up on the offer.

On November 18th, I'll be giving an introductory talk on InnoDB/XtraDB Performance Optimization. I will be the second speaker, with Carl Gelbart first speaking on Infobright.

What brings me to LA? On the same day (18th Nov) I'll be teaching a one day class on Performance Optimization for MySQL with InnoDB and XtraDB. If you haven't signed up yet - spaces are still available.

Entry posted by Morgan Tocker | No comment

Add to: | | | |

State of the art: Galera – synchronous replication for InnoDB

Vadim — Tue, 27 Oct 2009 15:08:58 +0000

First time I heard about Galera on Percona Performance Conference 2009, Seppo Jaakola was presenting "Galera: Multi-Master Synchronous MySQL Replication Clusters". It was impressed as I personally always wanted it for InnoDB, but we had it in plans at the bottom of the list, as this is very hard to implement properly.
The idea by itself is not new, I remember synchronous replication was announced for SolidDB on MySQL UC 2007, but later the product was killed by IBM.

So long time after PPC 2009 there was available version mysql-galera-0.6, which had serious flow, to setup a new node you had to take down whole cluster. And all this time Codership ( company that develops Galera) was working on 0.7 release that introduces node propagation keeping cluster online. You can play with 0.7pre release by yourself MySQL/Galera Release 0.7pre.

In current version propagation is done by mysqldump from one of nodes ("donor"). In next release Codership is going to support LVM snapshot and xtrabackup which will make the setup of new node even easier. The current annoyance I see is that if you shutdown one node for short period of time for quick maintenance, after start, the node has to load whole mysqldump, like it is new empty node. I hope Codership guys will address this also.
Another thing I miss for now is support of InnoDB-plugin, which as we know performs much better than standard InnoDB ®.

So what is so interesting about Galera. Couple things:

- High Availability. Any of N standby nodes are available immediately when main node fails. Galera is serious pretender to be included to the list, Yves put recently, http://www.mysqlperformanceblog.com/2009/10/16/finding-your-mysql-high-availability-solution-%e2%80%93-the-questions/. I am not sure how many nines it will provide , but efforts on test setup and deployment should be comparable with MMM setup.

- Scale Writes. Galera allows to write to any of N nodes and automatically propagate to other nodes. It sounds too ideal, and there is drawback - with increasing amount of nodes you write to, your transaction rollback rate may increase, especially if you working on the same dataset. You can find some results on Codership's page, and I am going to run my own benchmarks also. Also from benchmark you can see that communication overhead maybe significant for short writes.

- Scale Reads. It can be done with regular replication, but with synchronous your "slaves-nodes" are in the same state, there is no "slave behind". When you read from any slave, you read actual data. Although it also has serious drawback - our cluster is fast as fast the "weakest" node in the chain. So if one node gets overloaded and performance degrades, the same happens with whole cluster.

- Heterogeneous-database replication. It is not here yet, and I do not know what's in Codership roadmap, but group manager protocol in Galera is database independent, and it's only matter of database drivers. For InnoDB currently it is set of patches, and I see it is quite possible to make the same for Postgres. So MySQL-Postgres cluster setup is not so far ahead

On "Company page" Codership says their goal is "to promote and exploit the latest developments in computer science to produce fast and scalable synchronous replication solution that "just works" for databases and similar applications", which I think they have success in. Implementing fast, scalable and working group communication and transaction manager is the art.

As for now I would not put 0.7 release into production yet, but you may seriously consider to play with it in test environment, and report bugs to Codership team, they are very responsive.
I am waiting for next releases and looking to make integration with XtraDB.

Entry posted by Vadim | 11 comments

Add to: | | | |

XtraDB Amazon Image

Aleksandr Kuzminsky — Mon, 26 Oct 2009 04:36:49 +0000

For those who use Amazon EC2 service and were anxious about having XtraDB ready to launch there is a good news.

We created a public AMI (Amazon Machine Image) with XtraDB release 8 installed on CentOS 5.3.

How to use it.

First make sure it is avaiable.

PLAIN TEXT

CODE:

$ ec2-describe-images ami-4701e22e
IMAGE ami-4701e22e xtradb/centos-5.3-x86_64.fs.manifest.xml 834362721059 available public x86_64 machine
$

Run it. It is built for x86_64 plaform, so allowed types are m1.large, m1.xlarge and c1.xlarge

PLAIN TEXT

CODE:

$ ec2-run-instances ami-4701e22e -t m1.large
RESERVATION r-46b3432e 834362721059 default
INSTANCE i-ecc74084 ami-4701e22e pending 0 m1.large 2009-10-25T18:31:06+0000 us-east-1c

Wait till the instance starts

PLAIN TEXT

CODE:

$ ec2-describe-instances i-ecc74084
RESERVATION r-46b3432e 834362721059 default
INSTANCE i-ecc74084 ami-4701e22e ec2-75-101-203-143.compute-1.amazonaws.com domU-12-31-39-0A-26-22.compute-1.internal running 0
m1.large 2009-10-25T18:31:06+0000 us-east-1c

Now it is up and ready.

Entry posted by Aleksandr Kuzminsky | 7 comments

Add to: | | | |

Tuning for heavy writing workloads

Yasufumi — Thu, 15 Oct 2009 00:06:55 +0000

For the my previous post, there was comment to suggest to test db_STRESS benchmark on XtraDB by Dimitri. And I tested and tuned for the benchmark. I will show you the tunings. It should be also tuning procedure for general heavy writing workloads.

At first, . The next, to stabilize performance and to avoid decreasing performance.

Server:
PowerEdge R900, Four Quad Core E7320 Xeon, 2.13GHz, 32GB Memory, 16X2GB, 667MHz

db_STRESS:
32 sessions, RW=1, dbsize = 1000000, no thinktime

XtraDB: (mysql-5.1.39 + XtraDB-1.0.4-current)
innodb_io_capacity = 4000
innodb_support_xa = false
innodb_file_per_table = true
innodb_buffer_pool_size = 16G
innodb_read_io_threads = 8
innodb_write_io_threads = 8
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 128M
innodb_log_file_size = 512M
innodb_log_files_in_group = 2
innodb_max_dirty_pages_pct = 90
innodb_flush_method = O_DIRECT
(the followings are XtraDB specific general settings)
innodb_ibuf_active_contract = 1
innodb_adaptive_flushing = false
innodb_adaptive_checkpoint = estimate

At first, tuning the peak performance to use CPU and IO resource more effectively. To avoid mutex/lock contentions are good to use more CPU resource of many CPUs.

This graph shows the peak performance in tps of db_STRESS.

At current settings, "base" in the graph is the perfomance. We can confirm the mutex/lock contention roughly by the SEMAPHORES sction of SHOW INNODB STATUS output.

"xx-lock on RW-latch at 0x7f2ff40a3dc0 created in file dict/dict0dict.c line 1627"

It is index->lock, viewing the source file (and it may be HISTORY table). This is the lock for each index tree. We may be able to disperse the lock using by the partitioning of MySQL. Added the following clause to the HISTORY table definition.

"PARTITION BY HASH(REF_OBJECT) PARTITIONS 16"

Now the performance became to "+partitioned" in the graph. Looking the SEMAPHORES section again,

"has waited at handler/ha_innodb.cc line 7275 for 0.0000 seconds the semaphore:
X-lock on RW-latch at 0xd30320 created in file dict/dict0dict.c line 623"

may be the line which appears for the most times (it is dict_operation_lock). It may be partition specific lock contention. The current XtraDB has the variable to tune the contention.

innodb_stats_update_need_lock = 0 (default 1)

It skip the updating statistics which needs the lock. (it only affects for "Data_free:" value of TABLE STATS). And the performance became "+skip_stats" in the graph. Then, the next contention at SEMAPHORES section is...

"Mutex at 0x1b3e3e78 created file trx/trx0rseg.c line 167"

may be remarkable (it is rseg->mutex). The mutex is for each rollback segments, so we can increase the rsegs to solve the contention problem. XtraDB can increase the rseg.

innodb_extra_rsegments = 64 (affects to initialization of InnoDB)

Recreated database files with the parameter. Then the performance became "+rsegs64". At last, the next contention may be "Mutex at 0x28ce8e0 created file srv/srv0srv.c line 982". It is kernel_mutex, currently we don't have proper solution for that. The setting seems to be enough for now.

Next, looking the sequential result in more long term.

The next problem is "History list length" growing to huge size. The value is the number of entries in rollback segment. The entries are used for consistent reading of the older transactions. They can be removed when any transactions doesn't refer the entry. This removing operation for the entries is called "purge" in InnoDB. The purge operation should be done enough on time, because the huge history list affects to performance.

Basically, the purging is done by master_thread (general background thread of InnoDB). The huge history list makes the purge operation slow, and it interferes with the other tasks of the master_thread (e.g. flushing dirty blocks, treating insert buffer, etc...). Dimitri implemented a purge_thread to devote to the purging, and also XtraDB has similar purge_thread. Though it seems to make the throughput stabilize, it is not enough still for heavy update workloads. A single purge_thread on one CPU is not enough for updates from user threads on the all of other CPUs.

XtraDB can increase the purge_threads from the next release.

innodb_use_purge_thread = 4

seems to be enough for this workload on the server.

The first graph of followings is sequential throughput [tps] up to 3500 sec.

The next is tracking the "History list length" at the same time.

"Norm 1.0.4": Normal InnoDB Plugin 1.0.4 without XtraDB specific options
"xtra p_t 0": XtraDB 1.0.4-new (no purge_thread)
"xtra p_t 1": XtraDB 1.0.4-new (single purge_thread similar to Dimitri's)
"xtra p_t 4": XtraDB 1.0.4-new (4 purge threads)

The graphs show...

The purge thread (> 0) helps to stabilize the throughput greatly.
Increasing the purge threads can suppress the strong growing of the hitory list
The adaptive checkpoint "estimate" needs the purge_thread... (than the adaptive_flushing does)

And the last 300secs' average tps are...

"Norm 1.0.4": 5725.47
"xtra p_t 0": 4699.33
"xtra p_t 1": 7130.3
"xtra p_t 4": 9118    (about 60%up from Normal Plugin 1.0.4)

In the end, the faster and more stable performance of db_STRESS benchmark is obtained by these tunings of XtraDB.

-----------------------------------------

(Added 2009.10.29)

<FAQ: Is XtraDB slower than Plugin?>

I'd like to say "no" to this question. We have been adding many tuning options to XtraDB. But they are effective not for all cases, sometimes the performance may get worse because of "not proper" or "too much value". We should choose the options correctly. XtraDB is based on InnoDB Plugin and we can set XtraDB same to InnoDB Plugin at least. The following graphs are results of XtraDB and Plugin with same options and same database.

top-left (same condition to above graphs):
innodb_flush_log_at_trx_commit = 2
innodb_doublewrite = true

top-right:
innodb_flush_log_at_trx_commit = 1
innodb_doublewrite = true

bottom-left:
innodb_flush_log_at_trx_commit = 2
innodb_doublewrite = false

bottom-right:
innodb_flush_log_at_trx_commit = 1
innodb_doublewrite = false

It seems that XtraDB is not slower than Plugin here at least.

We can start tuning based on these performances using XtraDB specific options!

Why do you make XtraDB slower than Plugin?

Entry posted by Yasufumi | 9 comments

Add to: | | | |

Percona welcomes Yves Trudeau and Fernando Ipar

Baron Schwartz — Thu, 08 Oct 2009 18:21:53 +0000

I'm happy to extend a warm welcome to two new members of the Percona team.

First is Yves Trudeau, about whom I can say many things:

One of the top MySQL Cluster (NDB Cluster) experts in the world.
An expert on all things High Availability, including DRBD and Heartbeat.
Many years of experience with Huge Data.
Half of the Waffle Grid team.
A really nice person!

Yves joins us after a tenure of several years as a senior consultant at Sun/MySQL. Together with Matt Yonkovit, he plans to work on WaffleGrid (but as a new project under a new name, to be determined), and integration with XtraDB. Yves lives in Quebec with his family.

Next is Fernando Ipar. Fernando is our first dedicated Shift Support Captain^[1]. Fernando specializes in MySQL, GNU/Linux, systems administration, and high availability. Fernando has been involved in computer programming since he was 11, and in Open Source since 2001. He loves shade-grown coffee, music, and natural history. He lives in Uruguay with his wife and family. Fernando brightens our IRC channel with his helpful and cheerful presence!

Welcome, Yves and Fernando!

[1] Shift Support Captain is kind of a dispatcher for 24-hour service. We have been doing 24x7 service through an on-call phone number for years, but a few months ago we started literally manning the desk 24x7 -- something that we have grown large enough to both need and justify, at our current size of around 30 people. Our customers can now use email, Skype, AOL messenger, or Google Chat (as well as the phone number) to get immediate attention, 24x7, from a technically savvy person.

Entry posted by Baron Schwartz | 3 comments

Add to: | | | |

Which adaptive should we use?

Yasufumi — Wed, 16 Sep 2009 00:12:44 +0000

As you may know, InnoDB has 2 limits for unflushed modified blocks in the buffer pool. The one is from physical size of the buffer pool. And the another one is oldness of the block which is from the capacity of transaction log files.

In the case of heavy updating workload, the modified ages of the many blocks are clustered. And to reduce the maximum of the modified ages InnoDB needs to flush many of the blocks in a short time, if these are not flushed at all. Then the flushing storm affect the performance seriously.

We suggested the "adaptive_checkpoint" option of constant flushing to avoid such a flushing storm. And finally, the newest InnoDB Plugin 1.0.4 has the new similar option "adaptive_flushing" as native.

Let's check the adaptive flushing options at this post.

HOW THEY WORK

< adaptive_checkpoint=reflex (older method)>

oldest modified age %/max	behavior
0 ~ 50%	nothing
50% ~ 75%	constant flushing (weak)
75% ~ 87.5%	constant flushing (strong)
87.5% ~	(flushing storm)

< adaptive_checkpoint=estimate (newer method)>

oldest modified age %/max	behavior
0 ~ 50%	nothing
50% ~ 87.5%	estimate flushing as bellow *
87.5% ~	(flushing storm)

* estimate blocks to flush based on...

how many modified and unflushed blocks
progress speed of the transaction log
the modified age average of all blocks to flush

< adaptive_flushing (default "true" at 1.0.4)>
Its behavior is not based on the oldest modified age %.

* the how many blocks to flush based on...

how many modified and unflushed blocks
progress speed of the transaction log
transaction log capacity

(and adjust along with another flushing)

And it doesn't exceed the equivalent to "constant flushing (strong)"
of the "adaptive_checkpoint=reflex"

RESULTS

TPC-C like workload (100WH: 16 session full)

innodb_buffer_pool_size = 16G
innodb_max_dirty_pages_pct = 90

innodb_log_file_size = 512M
innodb_log_files_in_group = 2

innodb_io_capacity = 4000
innodb_read_io_threads = 8
innodb_write_io_threads = 8
innodb_flush_method = O_DIRECT

innodb_thread_concurrency = 0
innodb_ibuf_active_contract = 1

innodb_adaptive_flushing = false
innodb_adaptive_checkpoint = none

innodb_adaptive_flushing = true

innodb_adaptive_flushing = false
innodb_adaptive_checkpoint = reflex

innodb_adaptive_flushing = false
innodb_adaptive_checkpoint = estimate

case1: "innodb_doublewrite = false"

[0~1800 sec.]

[1200~1800 sec.]

averages (1200~1800 sec.)
none: 6868.92
flushing: 6655.92
reflex: 6481
estimate: 6575.88

case2: "innodb_doublewrite = true"

[0~1800 sec.]

[1200~1800 sec.]

averages (1200~1800 sec.)
none: 6569.48
flushing: 5090.12
reflex: 6871.9
estimate: 6609.9

CONSIDERINGS

The new "adaptive_flushing" seems to be tuned for "innodb_doublewrite = false" only.
(cause too much flushing for "innodb_doublewrite = true")

"innodb_adaptive_checkpoint = reflex" and "adaptive_flushing" need tuning innodb_io_capacity properly.
(The result is based on proper value "innodb_io_capacity = 4000")

"innodb_adaptive_checkpoint = estimate" is not depend on innodb_io_capacity, and it seems more "soft" result than the other methods.

So,

The "adaptive_flushing" seems good when we use "innodb_doublewrite = false".

"innodb_adaptive_checkpoint = estimate" may be safe for "innodb_doublewrite = true" for now.

I should adjust "adaptive_flushing" for "innodb_doublewrite = true".

IMPROVEMENTS (16 Sep.)

I have tried to improve the performance. But I couldn't adjust performance of adaptive_flushing with doublewrite... So, instead of that, estimate is adjusted more.

case1: "innodb_doublewrite = false"

[0~1800 sec.]

case2: "innodb_doublewrite = true"

[0~1800 sec.]

The new "innodb_adaptive_checkpoint = estimate" seems to be more stable than before. The next question is, "Why is estimate + doublewrite better than the other no doublewrite settings?"...

ADDITIONAL TEST IN ANOTHER BALANCE

Also tested in another configuration to check the behaviors in another balance of IO bound.

innodb_buffer_pool_size = 2G (1/8)
innodb_log_file_size = 128M (1/4)

case1: "innodb_doublewrite = false"

[0~1200 sec.]

case2: "innodb_doublewrite = true"

[0~1200 sec.]

"innodb_adaptive_checkpoint = estimate" seems stable but seems to flush much in its initial phase. The adaptive_flushing seems to be more soft in this cases.

Hmmm....

We should choose suitable adaptive_xxx for each workload for now...

Entry posted by Yasufumi | 12 comments

Add to: | | | |