July 26, 2014

FusionIO 320GB MLC benchmarks

After my previous benchmarks of FusionIO 160GB SLC card, FusionIO team sent me for the tests another card, FusionIO 320GB MLC. I should say I really appreciate an opportunity to play with this card and with combination of two cards.

This card is also not cheap, the price I can find on dell.com is $6,829.99, which is almost the same as for 160GB card, but with doubled space,
and which gives me 21$/GB for 100% formatted card, and 29$/GB for card with 75% of space allocated. Effect of 25% reserved space you will see in the results.


The full numbers are available on our Wiki benchmarks page, couple words about benchmarks.

I did sysbench fileio runs on 100% space allocated card and on 75% space allocated (240GB of
available disk space), and on 32GB and 100GB filesizes. After that I took two cards into software RAID0, and tried 100GB filesize in this configuration. Block size is 16KB in all cases.

The results for random write is here:
(Results are in MB/s (more is better) for random write test, pagesize 16KB )

threads32GB filesize, 100% space formatted 32GB filesize, 75% space formatted32GB filesize, SLC 100GB filesize, 75% space formatted 100GB filesize, RAID0 over 2 cards
183.917148134.33149.33132.91
4130.83418.88443.59411.02430.45
8177.34475.17566.54455.25692.58
16161.74488.64554.73472.3784.31
32150.9491.41457.6718.81
64146.08486.97387.17740.21

or in chart form:
fusionio,_random_write,_16kb

(if legend is unreadable, it is:

  • 32GB filesize, 100% space formatted
  • 32GB filesize, 75% space formatted
  • 100GB filesize, 75% space formatted
  • 100GB filesize, RAID0 over 2 cards

I hope Google Docs will fix the issue with font for the legend sometime
)

So there some numbers to consider:

  • With 25% reserved space you can get almost 3x throughput even with
    small enough files
  • SLC card gives about 15-20% more in throughput
  • With 25% reserved space, I was able to get 700 MB/s in sequential reads ( 400 MB/s in random reads) and 500 MB/s for random writes.
  • 95% response time in most cases are impressive, very often it is < 1ms (in hundreds microseconds range)
  • Sequential writes still behave strange, dropping throughput with increased concurrency
  • To get most from card you need multi-threaded IO (4-16 threads)
  • With RAID0 you can get doubled space with very interesting numbers, 750 MB/s in random reads and 800 MB/s in random writes ( and 1.2GB/s in sequential reads)

Now having these cards and numbers, I am going to benchmark MySQL with FusionIO as storage.

About Vadim Tkachenko

Vadim leads Percona's development group, which produces Percona Clould Tools, the Percona Server, Percona XraDB Cluster and Percona XtraBackup. He is an expert in solid-state storage, and has helped many hardware and software providers succeed in the MySQL market.

Comments

  1. HI Vadim,

    Great piece of hardware! Now this might be the fastest *single* storage device, but if you compare this solution to e.g. 8 Intel X-25M 80GB SSD’s attached to 2 Adaptec 5805 Raid cards you’ll get twice the amount of storage and more than double the read/write rates at less than half the price.

  2. Vadim says:

    Jos,

    There couple things to consider about 8 SSD cards

    1. I would be careful putting 8 cards into RAID 0 configuration. Most likely we talk about
    RAID 10, which halves storage space; or RAID 5 / 50, which hits write performance. With RAID0 setup
    will be quite unreliable.
    2. Where to store 8 cards. You need to come with your own enclosure box, or like that, which is pain
    if you have bunch of servers in datacenter rackspace.
    3. I have no data, but I think read/write throughput may be less than in FusionIO card, but I’d really
    like to test configuration you mention.
    I hope I get access to multiple SSD configuration someday.

  3. Hi Vadim,

    1. SSD’s are a lot more reliable than traditional disks; even if they ‘break’ you’d still be able to read the data off them. But agreed, RAID 0 if great for benchmarking but not for a production database
    2. It’s only 2 cards and 8 drives; the latter are only 2.5″ and two of them fit in one hard disk slot. No need for enclosures/racks, everything fits in a single box; in fact, with the right motherboard you could have 4 Adaptecs and 16 SSD’s in one 650Watt desktop enclosure.
    3. DM me at jostholis.com, I might be able to help ;-)

  4. Levi says:

    If you were running the X-25M/X-25E in production environment you would most likely keep it RAID-5 (reliability) or RAID-10 (performance/reliability). Doing the quick math on cost for parts (rounded values for ballpark figures):

    $7000 for FusionIO @ 320GB MLC
    $7000 for FusionIO @ 160GB SLC

    Also Intel recommends X-25E for server and workstation applications (plus it has higher write throughput and is a SLC design) so I have attached both price points:
    8 x $499 X-25M 160GB (MLC) ~= $2400
    8 x $999 X-25E 64GB (SLC) ~= $8000
    2 x $550 Adaptec 5805 ~= $1100

    Intel/Adaptec Solution SLC ~= $9100
    Intel/Adaptec Solution MLC ~= $3500

    X25M MLC RAID10 = 4x160GB ~= 640GB
    X25E SLC RAID10 = 4x64GB ~= 256GB
    X25M MLC RAID5 = 7x160GB ~= 1120GB
    X25E SLC RAID5 = 7x64GB ~= 448GB

    Considering the Intel solution is cheaper in the SLC and MLC design, I would be interested in seeing Vadim’s test (http://www.percona.com/docs/wiki/benchmark:fusionio:mlc:start) ran with a RAID10 and RAID5 configuration with the Intel drives and dual Adaptec cards. Both solutions are expensive, it would be interesting to see

    Also, has anyone started testing the High Point 4460?

  5. Shirish Jamthe says:

    Hi Vadim,

    Was Garbage collection running when the tests were taken? It will be interesting to know if these numbers were taken after a fresh format of the card or after several writes to the card. To make sure these are steady state performance numbers you may want to fill the card completely 3-4 times (overwrites) then immediately start the tests. Flash performance is a lot different when GC is running and the driver has to find good place to write while erasing the other prior used parts.

  6. Levi says:

    8 x $499 X-25M 160GB (MLC) ~= $3992 (Sorry was looking at the 80GB at $250)

  7. Vadim says:

    Shirish,

    I did benchmarks just after format. As understand Garbage collection activity is one of reason why
    FusionIO recommends 25% of reserved space, if you format card on 75% capacity, you should not see problems.
    However I am going to run long-running test to see how it performs.

  8. Andy says:

    Levi,

    In your cost calculation, if you’re mirroring Intel SSD for reliability, shouldn’t you also have 2 FusionIO in RAID 1 for an apple-to-apple comparison?

    In that case, the cost of FusionIO doubles:
    $14,000 for FusionIO @ 320GB MLC in RAID 1
    $14,000 for FusionIO @ 160GB SLC in RAID 1

  9. Fujipadam says:

    How about using just one X-25M SSDw ith remote backup and NO RAID. Arent they supposed to be more reliable than a normal HDD. With that the cost should be low and I would think the performance should be good too. (ofcourse I am far from an expert here)

    Fuji

  10. matt says:

    Solid State drives may be more reliable in the short term but, from my understanding based on a discussion with my brother-in-law who works for Micron (who supplies Intel), sold state has a tendency toward catastrophic failure.

    When solid state devices are created a certain amount of extra memory is put into the device for fail-over.
    As sections of memory die data is routed to the extra memory (that can not otherwise be accessed).
    Unfortunately when the extra memory is used and any other memory is lost the solid state drive completely fails.

    What I’m suggesting (and not 100% sure of since there could be some technology I don’t know about in SSD), is that the meantime before failure will generally be shorter in SSD drives than standard drives unless the SSDs are used primarily as READ devices (it’s writes that make sections of memory fail).

  11. matt says:

    Ooops, not Intel, IBM … lol

  12. >1. SSD’s are a lot more reliable than traditional disks; even if they ‘break’ you’d still be able to read the data off them. But agreed,
    > RAID 0 if great for benchmarking but not for a production database

    Jos,

    Tell that two the two 256GB OCZ Vertex SSD disks I have. Both failed within two weeks of one another, in different systems. Neither drive even shows up on the SATA bus when I connect the drives!

    One disk was connected to a adaptec dual core SAS controller, and the other to a regular old internal Intel mobo SATA II connector.

    I’ve honestly had serious performance or reliability issues with almost every SSD device I’ve owned and tested. I’ve gone back to plain old arrays of inexpensive disks, at least for my home stuff.

  13. Patrick Casey says:

    Matt, the “normal” failure mode that I’m aware of on SSD’s is the “lockng” of cells so that you can’t rewrite them, basically they get stuck. Net result is you end up with a read-only drive, which is still a failure, but its a “no data loss” sort of failure. I guarantee there’s other failure modes out there (Justin has an example), but, in theory at least, SSDs fail more safely than magnetic media.

  14. @Justin
    Tell me about it… I initially ordered 8 120 GB Vertex disks for my machine (that was the fastest solution *on paper*) but the Adaptec cards kept dropping the disks. Highly unstable combination. Luckily I could trade them in for 12 Intel 80 GB SSD’s, and that works fine. Also have an X-25M in my Macbook now; amazing!

  15. I’m going to look at picking up a few Intels when I have the budget for more expensive playthings. Testing out hardware products without the company sending them to you for evaluation is an expensive hobby.

    I’m thinking about creating a website dedicated to recording benchmark results. Users could register results for existing benchmarks, such as DBT3, SSB or sysbench, or to register their own benchmark. Vadim could register his ‘ontime’ benchmark on the site then record the results he achieved on his hardware, along with those hardware specs. Other users could run the same benchmark, and report their specs as well. This way, people could test different engines and different hardware and compare other results easily.

    Does this sound like something useful? Does something like this exist and I missed it?

  16. Vadim says:

    Justin,

    I’d love that idea on website with benchmark results, but each time I think about it even
    for publish my own results it seems very hard to implement properly..

    There are so many scenarios and results, so I can’t think how to formalize it.

    But if you have insights how to do it – I would like to see!

  17. These two sites have gobs of information about SSDs:

    – SSD news and history (some test reporting)
    http://www.storagesearch.com/

    – extensive tests (this link is to latest)
    http://www.anandtech.com/storage/showdoc.aspx?i=3667&p=1

    Points about SSD;
    - NAND is commodity; Samsung parts are good, others maybe less.
    - industrial strength drives (STEC, Texas, Fusion-io, and the like) have controllers which wear level the NAND such that one can write gigabytes/day and still last a decade or longer. Intel’s controller is still the most efficient in consumer devices. SandForce and Indilinx are catching up.
    - retail drives, Intel X-25M included, aren’t up to server needs. Pillar replaced Intel with STEC for this reason.
    - PCIe devices, such as the Fusion-io, aren’t really drives and aren’t bootable without a device driver. I recall reading that Fusion-io has a fix for Win7, but while they list a driver for RH/Suse and a build script for Debian/Ubuntu, I don’t see that this driver supports booting; it may.

    - Main Point: don’t bother using SSD if all you’re doing is running flat-file images in some database (MySql developers do this a lot). the real win with SSD is being able to run heavy join BCNF schemas, which are much smaller in footprint, and faster since they don’t contain anywhere near the volume of bytes.

  18. A database doesn’t have to be BCNF to do random I/O. InnoDB secondary indexes on a denormalized table can generate a lot of random I/O too. And while an SSD can improve join performance remember that all queries on MySQL are essentially single threaded (excluding for a moment the extra IO threads in XtraDB) so even with an optimal I/O subsystem there are plenty of other bottlenecks in joins such that denormalization will still almost always buy more performance. Also don’t forget that the optimizer may be forced to pick non-optimal join orders because of outer joins or hints, so just about any database which uses any joins can probably benefit at least somewhat from SSD.

  19. Trent Hornibrook says:

    Hi Vadim,

    Awesome post – I see that DELL has updated their page and are now selling the Duo drive:

    http://search.dell.com/results.aspx?s=gen&c=us&l=en&cs=&k=ioDrive&cat=all&x=0&y=0

    (Odd that the 640Gb is less than the 320!)

    Curious to see your results when you benchmark with MySQL.

    I’ve got my company looking at FusionIO now – just the cost ratio with consolidation against very expensive power and rackspace makes them attractive. Seems that for the price of 2-3 DELL 2950 with decent RAM we can get 2 FusionIO 320GB cards with the same spec’ed DELLs…

    -Trent

Speak Your Mind

*