July 25, 2014

Using XtraBackup on NFS for MySQL backups

XtraBackup works great for backing MySQL up to an NFS volume, but there is a gotcha that you need to be aware of. This applies to anything you do with NFS, not just XtraBackup. The gotcha is that NFS uses client-side caching to reduce overhead of sending data across the network.

If you backup to an NFS mount from your database server, and then use a different server that also mounts that volume to prepare the backup (apply the logs), that other server might not have a consistent view of the data. That’s because the data might not be flushed to the NFS server yet — it might just be sitting in the database server’s cache.

If this happens, xtrabackup will potentially see corrupt or inconsistent data. And InnoDB does not tolerate that. It will refuse to work, which is a good thing. The preparation step will fail. This is one reason why you should always prepare your backups so they are completely ready to use. (You should test your backups even more than that — we’ve blogged before about that. Let us know if you want help with that.)

You can solve this by mounting NFS synchronously on the database host. This is slower for general-purpose usage, but it should not be a big deal for xtrabackup. This and lots of other details about xtrabackup are documented on the XtraBackup docs.

About Baron Schwartz

Baron is the lead author of High Performance MySQL.
He is a former Percona employee.

Comments

  1. Chris says:

    What about using something like rsync for backups? What are the advantages/disadvatages for rsync vs nfs?

  2. NFS is popular because you don’t have to stage the backup locally first – you just mount the NFS volume where you like.

    (Edit: With xtrabackup, you don’t need to stage the backup locally however. It supports streaming.)

  3. A lot of people also have huge amounts of cheap disk space on their shared NFS servers. The disks they’re running their database on are much smaller and more costly. NFS makes financial sense for a lot of people. So does streaming the backup, as Morgan pointed out.

  4. Yingkuan says:

    Mount NFS with “noac” mount option should help in most cases.

    If XtraBackup can use O_DIRECT to copy files that will be even better :)

  5. noac only makes file and directory attributes (metadata) synchronous between client and server. It doesn’t make the file’s data synchronous, so it won’t prevent the problem. You need the noasync option instead.

  6. Brad says:

    for me, mounting NFS with the sync option didn’t fix anything. Instead I found and now use this:

    innobackupex –compress –stream=xbstream /doesntneedtoexist | ssh user@otherhost “xbstream -x -C /root/backup/”

    Just set up ssh rsa keys and you’re good to go.

  7. Daniel says:

    It’s entirely false that innobackupex –stream will somehow allow you to stream backups without staging them locally first. Xtrabackup stages its log file and *all* related data files locally before streaming, which can be massive. You are better off using NFS if you don’t have 2x space total available. Honestly, it’s not entirely clear why you would bother using the –stream option as it does anything but stream.

Speak Your Mind

*