April 17, 2014

InnoDB, InnoDB-plugin vs XtraDB on fast storage

To continue fun with FusionIO cards, I wanted to check how MySQL / InnoDB performs here. For benchmark I took MySQL 5.1.42 with built-in InnoDB, InnoDB-plugin 1.0.6, and XtraDB 1.0.6-9 ( InnoDB with Percona patches).
As benchmark engine I used tpcc-mysql with 1000 warehouses ( which gives around 90GB of data + indexes) on my workhourse Dell PowerEdge R900 ( details about box ).

On storage configuration: FusionIO 160GB SLC and 320GB MLC cards are configured in software RAID0 to store InnoDB datafiles. For InnoDB logs and system tablespace I used partition on regular RAID10 with regular hard drives, here I followed Yoshinori Matsunobu’s recommendations http://yoshinorimatsunobu.blogspot.com/2009/05/tables-on-ssd-redobinlogsystem.html and taking fact that FusionIO is not perfect for sequential writes http://www.mysqlperformanceblog.com/2010/01/11/fusionio-320gb-mlc-benchmarks/

Full results I put on page http://www.percona.com/docs/wiki/benchmark:fusionio:mysql:start, here are my thoughts and interesting facts.

First, chart with results for InnoDB vs InnoDB-plugin during runs (values are in new order transactions per minute, more is better) :

innodb_vs_plugin

As you see InnoDB-plugin is doing much better here, it allows to utilize multiple IO threads,
which as we saw ( http://www.mysqlperformanceblog.com/2010/01/11/fusionio-320gb-mlc-benchmarks/ ) is necessary to get most throughput from FusionIO.

Also you may see from graph some waves for InnoDB-plugin. Here we observe innodb_adaptive_flushing in action (which is ON by default), and I think innodb_adaptive_flushing in InnoDB-plugin is not quite balanced, it may do overaggressive flushing, when it is not necessary.

But looking on CPU stats (see graph later), I guess InnoDB-plugin spends most time in buffer_pool mutex, contention here is not fully resolved yet in InnoDB-plugin.

Now, let’s take XtraDB. In additional to multiple IO threads, we have patch to decrease contention on buffer_pool mutex, plus separate purge thread. Also we use different adaptive_checkpoint algorithm.

The results are:
innodb_vs_xtradb

So I guess buffer_pool improvements play here for XtraDB, and looking on summary result:

  • InnoDB 9439.316 NOTPM
  • InnoDB-plugin-1.0.6 15299.333 NOTPM
  • XtraDB-1.0.6-9 26160.551 NOTPM

InnoDB-plugin is 1.6x times better InnoDB, and XtraDB is 1.7x times better InnoDB-plugin.

Now on CPU usage and disk utilization.

Disk throughput:

disk_bo

CPU (user) usage:

cpu_usage

Even with improvements, XtraDB performs less 150MB/s in disk writes (from benchmarks we
saw FusionIO can do much more) and with 45-50% of idle CPU.

I assume we still see significant contentions inside XtraDB, and there big room for improvements. As for InnoDB-plugin, I’d wish InnoDB team makes some actions on buffer_pool mutex problem.

Finally I wanted to check what if we put innodb transactional logs and system tablespace on FusionIO also, there is graph for that:

innodb_logs_on_fusionio

It is not so bad, with final result 23038.283 NOTPM, it is only about 12% worse than with logs on separate partition.

And to make reference point, I run the same but with all files on RAID10 with regular disks,
the graph is there:

xtradb_fusionio_vs_raid10

with final result: 2873.783 NOTPM ( about 88% worse than all files on FusionIO)

To summarize

  • MySQL InnoDB/InnoDB-plugin/XtraDB is not fully able to utilize throughput of FusionIO. XtraDB is doing better job with internal contention, but much more can be done.
  • Still you can have very impressive performance improvement in IO-intensive or IO-bound workloads. You may want to use InnoDB-plugin or XtraDB to get better results.
  • Putting logs on separate partition may be good idea, but only if you have possibility to do that. Making special setup for that may be not worth improvements
About Vadim Tkachenko

Vadim leads Percona's development group, which produces the Percona Server and Percona XtraBackup. He is an expert in solid-state storage, and has helped many hardware and software providers succeed in the MySQL market.

Comments

  1. Magento says:

    It’s a difficult choices between XtraDB innodb & innodb plugin.

  2. dspezia says:

    Very interesting. Thanks for sharing this benchmark.
    Which I/O elevator did you use for the OS? Noop?

  3. Sadao says:

    tpcc-mysql shows the number of transactions processed every 10 seconds.
    So in these graphes, y-axis stands for not ‘NOTPM’ but ‘NOTP10SEC’ ;-)

  4. Vadim says:

    dspezia,

    Usually I used deadline (for RAID), but
    in this case there is no scheduler settings for software RAID nor for FusionIO cards.

  5. Vadim says:

    Sadao,

    Thank you, I fixed that.

  6. Do you think you would see any differences with the adaptive hash indexes disabled?

  7. David says:

    I think it is problem of adaptive flushing algorithms
    It’s not so aggressive to use the io
    set innodb_io_capacity=500 and disable the adaptive flushing to see the result

  8. Vadim says:

    Justin,

    I just made another run with the adaptive hash indexes disabled, but it does not help, and
    final result is worse.

    Actually in most cases, despite there is contention on adaptive hash index,
    disabling it does not help, as in this case, InnoDB has to do much more work on index pages scan.

  9. Vojtech Kurka says:

    Vadim,

    did you measure query latency? Throughput benchmarks look great, but the latency is crucial for web applications. We’re using XtraDB 7 with 40GB buffer pool, RAID10 with HDDs and when the write throughput goes up, we see read query latency dips in about 500-1500ms.

    What about the XtraDB RAID10 graph – the first part of the graph looks scary. Is there any explanation for that.

    Thank you for your great job and for publishing these benchmarks :)

  10. Vadim says:

    Vojtech,

    Unfortunately latency report from tpcc-mysql benchmark is rather limited, we may fix that one day.

    I have response time histograms for XtraDB and InnoDB-plugin

    XtraDB:

    1.New-Order

    (time,sec) (count of transactions)
    0.20, 1658223
    0.40, 2095
    0.60, 107
    0.80, 32
    1.00, 32

    InnoDB-plugin:
    1.New-Order

    0.20, 945959
    0.40, 6285
    0.60, 319
    0.80, 56
    1.00, 23

    It’s expected that increased throughput affects response time, from queue theory,
    the response time grow exponentially on increasing throughput.

    First part on RAID10 graph is just buffer_pool is warming up. I show results
    from 0 point of time (from restart of mysqld), and for RAID10 HDD it takes much longer to fill buffer_pool.

  11. Vojtech Kurka says:

    “First part on RAID10 graph is just buffer_pool is warming up.”

    I know, but I expected the graph to be much more linear. From this graph it seems the throughput touches zero all the time, until some volume of data gets to the buffer pool.

  12. Ben says:

    I’ve actually found xtradb SLOWER than innodb on my own benchmarks.

    I wonder if I’ve missed something with configuring Percona server?

    http://forum.percona.com/index.php/t/1623/

  13. Satya says:

    Hi,

    I did load test between Percona server and MySQL server using msqlslap

    mysqlslap -uroot -pimi@123 –query=”INSERT INTO sbtest3.sbtest VALUES(MD5(RAND()),MD5(RAND()),MD5(RAND()),MD5(RAND()),MD5(RAND()));INSERT INTO sbtest2.sbtest VALUES(MD5(RAND()),MD5(RAND()),MD5(RAND()),MD5(RAND()),MD5(RAND()));INSERT INTO sbtest3.sbtest VALUES(MD5(RAND()),MD5(RAND()),MD5(RAND()),MD5(RAND()),MD5(RAND()));delete from sbtest2.sbtest where a like ‘a%’;INSERT INTO sbtest2.sbtest VALUES(MD5(RAND()),MD5(RAND()),MD5(RAND()),MD5(RAND()),MD5(RAND())); delete from sbtest2.sbtest where a like ‘b%’;INSERT INTO sbtest3.sbtest VALUES(MD5(RAND()),MD5(RAND()),MD5(RAND()),MD5(RAND()),MD5(RAND()));update sbtest2.sbtest set b=’imi’;update sbtest3.sbtest set c=’imi’;INSERT INTO sbtest2.sbtest VALUES(MD5(RAND()),MD5(RAND()),MD5(RAND()),MD5(RAND()),MD5(RAND()));INSERT INTO sbtest3.sbtest VALUES(MD5(RAND()),MD5(RAND()),MD5(RAND()),MD5(RAND()),MD5(RAND()))” –concurrency=50 –iterations=100

    Table structure of sbtest2.sbtest is
    CREATE TABLE `sbtest` (
    `a` varchar(100) DEFAULT NULL,
    `b` varchar(100) DEFAULT NULL,
    `c` varchar(100) DEFAULT NULL,
    `d` varchar(100) DEFAULT NULL,
    `e` varchar(100) DEFAULT NULL
    ) ENGINE=InnoDB DEFAULT CHARSET=latin1

    Table structure of sbtest3.sbtest is
    CREATE TABLE `sbtest` (
    `a` varchar(100) DEFAULT NULL,
    `b` varchar(100) DEFAULT NULL,
    `c` varchar(100) DEFAULT NULL,
    `d` varchar(100) DEFAULT NULL,
    `e` varchar(100) DEFAULT NULL
    ) ENGINE=InnoDB DEFAULT CHARSET=latin1

    In this case, percona server takes more time than MySQL server.

    Percona : 9:35 min
    MySQL : 8:02 min

    Is there any problem in my load test?
    Please suggest me.

Speak Your Mind

*