Comments on: Why you can’t rely on a replica for disaster recovery

By: crownedgrouse

crownedgrouse — Thu, 07 Oct 2010 14:53:04 +0000

Hi,
A replication is not a backup. It is aimed to provide High availability (HA) , not disaster recovery (DR).

To be safe (the minimum !) :
1) use good harware. Mainly by doubling all devices (power, etc …).
2) use RAIDx disks on a SAN (be careful on the ‘write back’ or ‘write through’ parameter !)
3) use ZFS if possible, or at least a journalized FS.
4) use ACID compliant database (postgresql …), use internal replication feature if possible, for HA and DR (see 6)
5) snapshots can be usefull sometimes, but rarely for databases because data can be inconsistent.
Don’t snapshot database directories, except if you can tell the database that you are backuping or snapshoting the data. (Postgresql allow it : SELECT pg_start_backup(‘label’); )
ZFS allow powerfull snapshots that can be sent to another remote node…
6) do regular database dumps in a (compressed) file for snapshots and backups. That can be done on a replicated slave database for performance.
7) do complete and incremental backups on tapes, disks and optical disks. Protect it in fireproof vault.
use system versioning for system configuration (/etc/, …) and protect it (tripwire …)
9) duplicate the whole in two datacenters in two different cities, at least, or two continents if possible.
cross the backups : Datacenter A is backuped on datacenter B and the contrary.
10) test data recovery and your disaster recovery plan (imply you wrote it :>) …)

And pray… if you are believer.

By: Baron Schwartz

Baron Schwartz — Mon, 16 Aug 2010 18:13:05 +0000

InnoDB does support raw devices, and if used, then this type of problem is avoided. But it’s kind of nice to be able to use “cp” and “mv” and other tools. Most people don’t want to give that up. “Complete and total control” comes with limitations.

By: Andrew Watson

Andrew Watson — Mon, 16 Aug 2010 17:59:03 +0000

Why doesn’t MySQL support raw devices like Oracle? Then you don’t have to rely on a file system driver for anything! you write the bits out to the device yourself and have complete and total control over that device.

By: Baron Schwartz

Baron Schwartz — Mon, 16 Aug 2010 12:51:18 +0000

Read previous comments, please.

By: Andreas

Andreas — Mon, 16 Aug 2010 10:51:40 +0000

InnoDB got a transaction log. Isn’t this log responsible for recovering any glitches after a power outage?

By: Baron Schwartz

Baron Schwartz — Mon, 16 Aug 2010 10:45:22 +0000

The unreliability of ext2 is just one specific example of a larger principle, in my opinion.

By: Florian Haas

Florian Haas — Mon, 16 Aug 2010 09:48:50 +0000

I know I’m a bit late in responding to this, but isn’t this really about “why you can’t rely on ext2 for anything”? I think pretty much everyone agrees that using a non-journaling file system is a really bad idea if you want any resilience in the face of power outage at all, regardless of whether any sort of replication is involved.

Just my $.02.

By: Baron Schwartz

Baron Schwartz — Sun, 08 Aug 2010 03:05:33 +0000

Given a properly behaving operating environment, InnoDB does not corrupt your data, and it is better than most at detecting and recovering from many types of misbehavior. But if the database writes and fsync()s a file, and the filesystem botches the result good and proper, there is nothing the database CAN do.

For a different perspective on this topic, I suggest reading http://www.xaprb.com/blog/2010/02/08/how-postgresql-protects-against-partial-page-writes-and-data-corruption/.

By: Patrick Casey

Patrick Casey — Sun, 08 Aug 2010 01:37:18 +0000

Andreas, I think it actually depends on both the DB and the fileystem. The most common linux file systems like ext3 are journaling file systems and as such they very, very, very rarely get corrupted during a hard power off, but it does happen. If your file system gets corrupted, it really doesn’t matter what kind of shape the database is in because the underlying FS is munged and you can’t get your blocks back.

In practise, I can’t recall a snap of mine ever failing a recovery on EXT3, buts its not outside the realm of possibility.

Point being I suppose is that modern file system almost always survive a crash, so we tend to take that kind of durability for granted, but it really isn’t. The fact that a server comes back at all after an abrupt power loss is a testment to some very careful file system design. The fact that a database running on top of that file system recovers as well is still more good engineering, but both have to work to get your data back .

By: Andreas

Andreas — Sun, 08 Aug 2010 01:03:39 +0000

Quote: A snapshot on the SAN is just the same as cutting the power to the machine â€” the block device is in an inconsistent state.

So you are telling me i can loose all my data just by cutting power?
I won’t blame the filesystem here, i will blame the database.