This is ordinarily the LAST thing I would recommend, because I tend to think there’s always more performance that can be squeezed out of most environments – but in this case the only sane solution is to apply more hardware.

Regargind your two items – as you can see they do not have much to do with performance but rather how do you have you operations and high availability setup.
I dislike SAN mainly because their cost. For most high volume applications the difference is just too high.

In reality if you look at the physics it all comes down to amount of disks you have, cache you have etc. The “smart software” can optimize things sometimes a bit bit you should not expect any magic, in particular free magic – many optimizations are helpful for some workloads but bad for others.

Regarding snapshotting LVM works pretty well if it is setup properly. You have to plan certain IO capacity for it but it is still typically more efficient.

About loosing the server… I prefer to design systems there is never dependency on single copy of data. You can get tables corrupted because of MySQL bugs (or filesystem, hardware etc) – so I would rather keep the slave or DRBD partition on the other node which can be used for failover.

Though there are a lot of choices and conditions, among them some people just love SANs so purchasing them may make sense to keep them happy at very least

We tested pretty extensively with RAID 5 arrays (5 disks in the array if I recall), but we could still IO it out if there was a spike of some sort, plus with that kind of array attached, the boxes weren’t all that cheap anyway.

You’re definately right about the memory though, we’re collapsing 4 8G boxes into 1 32G box, so there’s a non linearity in cost there.

The other nice thing we liked about the SAN was:

1) Snapshotting was very fast and efficient
2) If I lost a blade, I could just mount the LUN on the hot spare and be up again in a matter of minutes

Again though, those are advantages we paid for (san != cheap).

What kind of storage did you have on these boxes ?

My general experience is SAN/NetApp are pure solution for MySQL Server when it comes to Performance. It can make things easy for you to manage or deal with cases when it is just hard to get so much of local storage.

IO system starvation can be the problem for sure but good directly attached storage is often good answer for that.

Think about Caching too. Collapsing 4 boxes into one you need 4x more memory to maintain the same hit ratio and so to have just 4x IO subsystem demands.

The question of the balance is also always the good question. I often see people reading word “commodity” as “cheap crap”. So they end up buying very cheap boxes for databases – little memory, no good RAID, poor hardware components and when loose a lot of time managing these.

To give the background here, we deploy a LOT of mysql running a not entirely predictable workload (we allow freeform query in our app). We started out buying lots and lots of what my operations staff calls CABs (Cheap Ass Boxes). We did some analysis though and demonstrated that we were “wasting” a lot of those boxes.

We’d have 4 or 5 boxes, for example, with CPU loading under 20% and healthy looking memory, but we couldn’t collapse them onto one box because if we did that the IO subsystem would lock up and performance would be terrible as soon as anybody did a table scan.

It turned out it was cheaper for us to invest in some high end storage (a Netapp SAN) and replace 4 CABs with one mid teir blade with a lot of memory.

Net effect is we have roughly the same cost structure, the same or better performance, and 1/4 the number of boxes under management.

Has anybody else tried to solve the same problem on a large scale? I know that investing a mess of money in a SAN isn’t the mysql “way”, but I’m kind of curious as to how other folks are solving the IO problem (which admittedly isn’t mysql specific).