August 2, 2014

FlashCache: first experiments

I wrote about FlashCache there, and since that I run couple benchmarks, to see what performance benefits we can expect.
For initial tries I took sysbench oltp tests ( read-only and read-write) and case when data fully fits into L2 cache.

I made binaries for FlashCache for CentOS 5.4, kernel 2.6.18-164.15, you can download it from our testing stage. It took some efforts to make binary, you may get my instructions for CentOS on FlashCache-dev mail-list, most likely it will not work for different CentOS / Kernel.

The full results, scripts and settings are on Benchmarks Wiki.

* Hardware: Dell PowerEdge R900
* IO subsystems
* RAID10: 8 disks SAS 2.5″ 15K ( main storage)
* SSD-1: Intel X25-E 32GB ( 1 gen firmware) ( Model Number: SSDSA2SH032G1GN INTEL, Firmware Revision: 045C8621)
* SSD-2: Intel X25-M 160GB ( 2 gen ) ( Model Number: INTEL SSDSA2M160G2GC, Firmware Revision: 2CV102HA )
* FlashCache: build over SSD-1 or SSD-2
* Filesystem XFS, builds as: mkfs.xfs -f -d su=16384,sw=40 /dev/sdc
* mounted with -o nobarrier option
* InnoDB files layout: ibdata1 and ib_logfile* are placed on separate RAID partition ( not on FlashCache or SSD)
* Benchmark: sysbench oltp ( read-only and read-write modes), 80mln rows (~18GB of data)

And I am comparing results with data stored:

  • Directly on RAID
  • Directly on SSD
  • Data stored on RAID, but caching in FlashCache ( which located on SSD)

The results for Read-Only case:
read-only

As expected with FlashCache the results are very close to running directly on SSD, with small drop, which could be related to additional driver layer.

Read-Write case is more interesting. There I should mention that FlashCache uses WriteBack caching algorithm, which keeps some amount of dirty pages before physically writing to main storage. FlashCache allows you to set target percentage of dirty pages, so it was interesting to see how it affects ( expected that smaller amount of dirty pages puts more IO on main storage leading to smaller performance).

So there results for FlashCache with 20% and 80% of dirty pages and FlashCache based on Intel X25-M:

Read  Write X25-M

So it is pretty much impressed with 20% dirty pages we still have decent improvement.

And there are results for Intel X25-E:
ReadWrite X25-E

By some reason the results much worse if compare to X25-M, probably X25-M in second generation has better performance characteristics ( I will post sysbench fileio benchmarks). Also what is attractive in X25-M, that even in biggest capacity 160GB you can get it by quite acceptable price.

In general FlashCache leaves pretty good impression, I did not have any significant problems ( not speaking about crashes or data loss), probably, the most challenge is to get a binary driver for your kernel :). I will do more benchmarks especially when data exceeds memory, and also it will be interesting to see how it works when we put FlashCache on FusionIO cards instead of Intel SSD.

About Vadim Tkachenko

Vadim leads Percona's development group, which produces Percona Clould Tools, the Percona Server, Percona XraDB Cluster and Percona XtraBackup. He is an expert in solid-state storage, and has helped many hardware and software providers succeed in the MySQL market.

Comments

  1. Andy says:

    Did you use BBU for your RAID 10?

  2. Vadim says:

    Andy,

    Sure, RAID10 is with BBU .

  3. peter says:

    Vadim,

    It may be interesting to compare X25-E to X25-M partition of the same size. The larger partition can cause significantly different cache behavior even if data fits in it completely.

  4. Dimitri says:

    Vadim,

    any details about which part of SSD was used for FlashCache? (whole disk? partition? etc.)

    Until the SSD may contain all data – it’s clear it’s more simple to move your database(s) to SSD.. But what if no? – it’ll be interesting to see how well FlashCache may improve performance if it may contain only 10% of data (for ex.) – likely what will be the price/performance ratio of placing one 32GB SSD as FlashCache into existing 320GB database, comparing to replacement by x10 32GB SSD..

    Rgds,
    -Dimitri

  5. Domas says:

    Dimitri, as with any other I/O based benchmark, flashcache performance improvements depend on data distribution. If you have long tail but very hot data at the head of it, it will help. If your access pattern is completely random – not so much.

  6. Dimitri says:

    Domas, completely agree :-)
    that’s why it’s important to know the worse case..

    On the same time it’ll be fine to have an easy way to monitor the data access distribution in general, and then predict a possible performance gain..

    Rgds,
    -Dimitri

  7. Another hack is to mount the SSD as swap and tell InnoDB to use say 100GB of memory. I haven’t tested this but it might be a fun hack :)

  8. markt says:

    Very interesting, thanks for the hard numbers!

  9. Joerg M. says:

    I don’t know if this test is really realistic, as it delivers performance without guaranteeing consistency:
    - usage of the nobarrier option
    - using SSD without battery cache protection as write-back cache

    Without protected cache, i would suggest 0% dirty pages…

  10. Vadim says:

    Joerg,

    Your points are totally correct, I asked FlashCache dev if we can use WriteThrough more instead of WriteBack, as even dirty_pages=0% still do not warranty full consistency.

    We may also disable write cache on SSD (something I am going to test) or use some mirroring for SSD level.

  11. Vadim says:

    Dimitri,

    I used whole partition for SSD.

    Sure I am going to test cases when data exceeds SSD capacity.

  12. Vadim says:

    Kevin,

    I tried that.
    IMO Linux kernel is not able to work properly in that mode.

  13. Mark Callaghan says:

    The Facebook patch for MySQL supports that. FlashCache is much better.

  14. darkfader says:

    80% dirty rate and barriers off so it’s “fast”…

    Err, good you also ran some benchmarks that apply to people whose data is more value than social networking or online store.

  15. Clemens Eisserer says:

    It would be really interesting to see how flashcache copes with barriers enabled – as for many reasons I would not be comfortable barries disabled.

    With flashcache beeing copy-back, the relative performance gains should be even better =)

  16. Lee says:

    You said
    “* InnoDB files layout: ibdata1 and ib_logfile* are placed on separate RAID partition ( not on FlashCache or SSD)”

    Why did you exclude ibdata1 from flashcache ?
    I think flashcache is good place to store ibdata(innodb system tablespace).

    And, Is Xtrabackup compatible with Flashcache ?

    Thanks.

Speak Your Mind

*