Thank you for sharing results. They are in line with my experience.
With RAID controller, ‘cfq’ is not suitable to use.

Very short recap with some numbers (40 simultaneous seq reads):
XFS+noop: 140MB/s
EXT3+noop: 133MB/s
EXT4+noop: 111MB/s (exploded once with corrupt data when changing scheduler)
EXT2+noop: 97MB/s
XFS+cfq: 30MB/s

Some comments on this setup:
- XFS beats EXT*, with EXT4 performing badly and also being unstable
- cfq vs noop makes a *lot* of difference. In this setup cfq seems close to broken. deadline is similar to noop.

http://www.fishpool.org/post/2008/03/31/Optimizing-Linux-I/O-on-hardware-RAID

(read the last part of the article).

With <2.6.24 kernels I even experienced a bug in which both MySQL and the job were running ridiculously slow with barely no IO operations on the RAID.

Might be different for other subsystems, but for be deadline just rocks, even without BBU (6-drive Raid10 though)

And sadly no, theres no per request priorities yet as far as I know (not even per posix thread yet, but just per process, which indeed is way more pratical to use for PostgreSQL than it is for MySQL right now, but that may deserve a small little smallish tiny fork, maybe . Same goes for ioprio_set(2).

Did Linux get per request priorities finally ? I worked with an University some time ago to see how IO priorities can be useful and they were quite helpful allowing to work with large queue sizes without starving log writes etc.

Now about disk IO scheduling and cache. Really it is a bit different though connected. The dirty pages to be flushed are picked by the kernel (background flush or forced by fsync) and when submitted to the kernel together with reads request. If you have a lot of dirty pages there is indeed a better chance to optimize IO.

Yeah XFS rocks in terms of not serializing O_DIRECT writes and also does not have contention problems on meta data updates
http://www.mysqlperformanceblog.com/2009/01/21/beware-ext3-and-sync-binlog-do-not-play-well-together/

The problem is most people run on EXT3 and it is hard to make them change

Domas,
does this differs from ext3’s data=writeback side effects?

Peter,
my comment about O_DIRECT w/o BBU was based on the (very naive and unverified) believing that Linux I/O schedulers do reorder based on kernel’s page cache content (so, it seems I was wrong), and on the fact that software based reordering matters when one don’t have hardware to handle it, MySQL using O_DIRECT to move data from sequentially written log files to the more write-costly spreaded/random innodb data file. Whatever, I stand corrected now, thank you

You are also very on point when you bring up latency in the discussion (and also queues size/depths); indeed scheduling is not only about aggregating write requests and bulk performances, but also about arbitrating resources usage among consumers and making decisions about avg vs. worst case latencies vs. throughput, which sometimes does matters.

I suspect a mixed workload (alternated write intensives, reads, and cpu bound moments, not just huge sustained writes) would give controller better chances to reorder properly before cache being filled up. If so, scheduler added complexity may be even more a waste, but in those conditions, user’s perceived performances may be more reflected by per-request time to completion (so, latency). I mean workloads including possibles concurrents reads and cpu work, idling moments, etc. as opposed to steady, bulk, write-only workloads (ie. importing dumps), where one may rather account the total, wall clock time needed to write the whole dataset without interruptions. Having a nicely interactive ajax interface stuck because some bulk thing does fill the BBU cache, the I/O queue and the innodb log file in the background may not be the expected behaviour, even if it gives a lower cumulated time to complete (therefore more TPS).

On this matter: thanks to the recent work on cgroups and containers resources control, Linux got a lot of refining in order to expose and enforce I/O priorities and arbitration. I’m not sure whether MySQL do leverage those features though (maybe it would be useful to lower priority for logs->data moves, or to increase priorities when big locks are held, or when deadlocks starts to happen?) neither how well do the different I/O elevators handles those features. And to the best of my knowledge, SQL does lack a way to specify expected QoS per request (I mean, something like TCP/IP TOS field) which would allow exposing this to consumers (although MySQL offers INSERT DELAYED).

Back to the subject, I don’t know anything about tpcc (and even less about how it’s used here): does this bench really spreads writes to emulate some randomness (wrt block device layout) or is it more of a “plain sequential bulk insert/update on a very few tables” thing?

Seekwatcher (http://oss.oracle.com/~mason/seekwatcher/) graphs would be interesting for such benchmarks, if anything to show what patterns you actually do measure (that’s to insidiously revive the “Tools” post .

And there’s no difference between noop and deadline is simply because deadline is noop with starve protection. If you do not starve too much, there’s not much need for deadline.

An interesting note about O_DIRECT though… and so how it can affect results. In this kernel (at least) O_DIRECT serialized writes on inode level which can affect things significantly.

I also did not quite get the point as without BBU and so without O_DIRECT – even if you do not have BBU O_DIRECT is often very helpful allowing to avoid double buffering. It may be slower though in case workload is write bound because IO serialization for O_DIRECT.