Using MySQL Enterprise edition is not going to change anything.

Could it be that mysql enterprise edition has something for these compared to the community version?

Please have a look at Tungsten Replicator. We have solutions for the consistency problems that tend to break MySQL replication (your point #2). These include global transaction IDs for safe slave promotion, crash-safe slaves, and built-in consistency checks similar to mk-table-checksum. Also, we will be introducing parallel replication during the next quarter, which is one answer–and a good one–to your point #1 about slave lag.

Cheers, Robert

p.s., Not to disagree at all with your conclusions about DRBD. For the right cases it really can be the cat’s miaow.

I would mention few things

1) When you want application to be up often does not only applies to failures but also minimizing scheduling downtime needed. MMM allows much more here – ALTER TABLEs, MySQL Upgrades, Storage Engine changes etc.

2) Anders: The switch to the slave is not instant – it indeed needs to catch up but in well configured systems (in particular designed to minimize switch time) the lag should be fractions of the seconds.

3) Anders: Indeed it is possible for master to remain in tact after failure. It would be reasonable to run mk-table-checksum before putting traffic back on it or simply reclone it from the slave using MMM. You also may resolve the changes at the same time.

4) Yves: The trick is NOT to let your slave to get lagged behind. This indeed requires more discipline – if you use DRBD all overhead is on the Master so it simply can’t handle the load which means it gets instant attention while replication lag may be allowed to appear.

5) Yves: Yes… Replication is a risk here. The monitoring for replication of course must be in place as well as regular checks with mk-table-checksum. In general for most applications replication can run rather stable.

“How do you failover if the surviving master is lagging 2 hours behind”

You don’t let it lag 2 hour behind. I know this is easier said than done (with replication being single threaded), but this would be one of the requirements for replication based availability.

“Replication often breaks, while broken, HA is very limited.”

That’s true. But this comes down to the same argument in my post – “MySQL Replication (with a manager such as MMM) aims to have 100% availability”. In this context availability means purely “Uptime” not redundancy.

“Since Master-Master rely on replication and replication as no built-in self-healing, it is my opinion that a Heartbeat/DRBD HA solution is easier to manage, especially in shops where there are no full-time DBA.”

Ease of use is always a difficult/subjective component to measure – especially when it can get in the way of choosing the best (performance-wise) solution. I don’t dispute how helpful the built in functions of DRBD are (online device verification, replication event checksums, fast resync after a node has been offline), and I wish that MySQL Replication had the same features. But it’s fairly easy to emulate most of what you need with tools like maatkit and mmm.

With Master-Master, here are my 2 big cons:
- How do you failover if the surviving master is lagging 2 hours behind?
- Replication often breaks, while broken, HA is very limited.

To be honest, I must also add a con on the DRBD side, write performance is slower. Depending on the hardware, it can be significant, a 30% drop is not uncommon. Most of the time, it is due to the networking connection between the two nodes, even if bonding is used to boost the bandwidth.

Since Master-Master rely on replication and replication as no built-in self-healing, it is my opinion that a Heartbeat/DRBD HA solution is easier to manage, especially in shops where there are no full-time DBA.

* MMM acts as the HA framework. On a failover, it will wait for the relay log to be empty on the slave.

* For failback, it’s true the master can’t be trusted straight away, but as Peter mentioned mk-table-checksum is your friend

* The ‘dual lag problem’ is also handled by MMM – it will monitor that the slave is not greater than a certain number of seconds behind the master, and alert you if it is (this problem relates to bullet point #1).

- Morgan

I think this is an issue that needs to be addressed by the MySQL Server team, since not only does it make failover with DRBD more difficult, but it makes server restarts to turn on features like ‘log-bin’ very painful.

Jeremy provided a proof of concept patch to one possible solution:
http://provenscaling.com/blog/2008/10/06/making-mysql-more-usable-innodb-saverestore-buffer-pool-patch/

If the idea he mentioned “Add runtime commands to save/restore the buffer pool state from a running server, without shutting down” was implemented, this could be used for DRBD to periodically checkpoint.

@Jeremy: I understand people commonly “preheat” caches on MySQL startup using a clever SELECT script with –init-file, so as far as I can see that point is moot — am I mistaken?

@Anders: I’d love to see someone write a Master/Slave capable, MySQL replication aware version of the Pacemaker mysql OCF resource agent so people could get the best of both worlds while using the same cluster framework all the time. I’ve tried myself; but found myself lacking the insight into MySQL replication to implement this properly. Any takers?