It is tricky. If you have single drive 100% utilization as reported by iostat actually means drive is 100% busy because there is at least one outstanding request all the time. However even with single drive having multiple outstanding request allows drive to execute them in better order and thus driving performance higher than it was when it was 100% busy with only one request.

When you get to the RAID it becomes very complicated because now you need multiple requests in parallel to load the system and they have to come to the right drives – otherwise some drives may have capacity while others are pegged.

In practical terms I look at “wa” in VMSTAT, iostat -dx utilization rate as well as service time and average time in queue. When IO system becomes overloaded the average time in queue can increase dramatically.

This is one thing I’ve always been curious about: in your post you mentioned “100% IO subsystem utilization.” Is this a theoretical number, or is it possible to determine the exact percentage of IO utilization on a Linux system? Is it something you can gather from the iostat command?

I am not as current as I would like to be with NDB Cluster, but certainly more memory would allow more resources for transaction and query handling. Look out for configuration parameters with a prefix of MaxNoOf like MaxNoOfConcurrentOperations. Increasing the number of replicas will not increase speed, however increasing the number of nodes and therefore fragments per partition should increase speed.

The NDB engine supports data on disk and in memory.
http://dev.mysql.com/doc/refman/5.1/en/mysql-cluster-disk-data.html

The important factor to consider when upgrading memory on an NDB Cluster is that the data nodes all need to run with the same memory specification, therefore any memory upgrade must occur across all data nodes.