Something that I think a lot of people / companies overlook is power protection. I’ve seen massive clusters plugged directly into standard outlets (Gulp)… Invest in a good UPS system if for nothing else than to bring your servers down nicely, it really goes a long way to preserve hardware.

Not sure how people keep their jobs when they lose hardware every time the lights flicker, but there’s a ton of companies doing it.

Indeed DRBD is one of the good approaches to MySQL HA and is especially good one if you can’t afford loosing any transactions but do not mind several minutes downtime (and some slowness while you warmup cache) as you switch. Others choose to relay on MySQL replication (sometimes with google Synchronous replication patch) which has fast recovery and less warmup problems but has a bunch of problems on its own.

This makes sense if you happen to have “low end” boxed. There are many clusters especially for high growth applications which got to this stage quickly which have pretty top end commodity hardware. But I guess we understand each other.

I would be focusing on Price/Performance for the choice however the price should also include man power, power, space etc. In this case cheap low end boxes will often be loss.

If your cluster is < 50 machines you’re probably better off scaling diagonally vs scaling horizontally / vertically.

Inexpensive does not mean cheap It means commodity more than cheap in this case.
Same as with RAID – you’re not saying because it says about Inexpensive you have to have your RAID filled with cheapest SATA drives ? You quite well can do with SAS drives which are more expensive but still.

Also number matters – it may be for your application $20K servers are best choice, while for other it is $5K Servers – as long as you have many of them and you’re not relaying on all of them staying up it is fine

*inexpensive* servers implies that you’re buying the cheapest boxes imaginable.

It’s really a “bang for your buck” computation.

We’re using somewhat mid-level machines now which has been yielding significant costs savings both in terms of machines pricing, power, and administration costs.

Kevin

Indeed disk is the most typical component to fail. Speaking about slaves – indeed it is easy. It is easy to load balance transparently so you do not even notice if slave is gone and it is very easy to re-clone slaves automatically from the other slave or master. What people are typically concerned is master failures

I would really like more availability shared by people which run big amount of boxes. I think you’re right about human error and here a lot depends on automation and how uniform system is. If you have 100 different systems it is much easier to screwup compared to running 1000 of instances of “sharded” data when you only use some toolset to do global operations (because it typically has to be done on all of them at once and because it would be a pain to do it manually)

We use DRDB with a spare master to approach MySQL HA. Although many people feel like it is a waste of a perfectly good server, we simply can’t afford to wait an hour (or more) for a technician at the datacenter to realize our primary server is failing, find the problem, replace the part, wait for the machine to reboot, and so on. As with all scalability plans this one is probably not permanent, but for now it works well enough and fits within our budget.

Mike, I’ll definitely lament with you that my #1 source of downtime is running a query that I shouldn’t have. With large data sets under high load, sometimes it can be difficult to understand the full consequences of queries that in other circumstances would not be an issue. My colleagues worry about downtime while I’m on vacation, but they should worry more when I’m at my desk tinkering