Comments on: Storing MySQL Binary logs on NFS Volume

By: Peter Zaitsev

Peter Zaitsev — Mon, 18 Apr 2011 17:41:13 +0000

William,

In this case the point is rather the network roundtrip seems to be needed (at least in Linux client implementation) if file is to be extended. Your last sentence says it all though. In many cases people think moving to high end NFS will get far better performance than local storage.

By: William Jimenez

William Jimenez — Mon, 18 Apr 2011 17:31:47 +0000

What type of file system was on this NFS server? Remember that NFS sits on top of the file system residing on the server you are exporting the share from. While I agree that NFS can be tricky, these aren’t plug and play technologies so the makeup of system stack holistically has a huge impact on the performance.

NFS is one of those technologies that either performs very well or very poorly, depending on how you configure it. I understand that the point of this article is not to be advice as much as a discussion starter, so I am just adding some more color to the picture .

ZFS based NFS on Solaris based kernels have had very impressive results, something to look into. You won’t see anything like local or direct attached storage however….

By: peter

peter — Tue, 03 Aug 2010 23:18:37 +0000

Copying Binary logs after they rotate is easy enough. If you use NFS/GFS/DRBD volume for storage logs they are immediately available after Master crash which allows you to use that logs to bring Slave up to speed and have no transactions lost if you have sync_binlog=1

Indeed the local storage is a lot faster you often can get over 10000 fsync/sec with local RAID while only some 3000 with NFS other 1GB ethernet.

By: Baron Schwartz

Baron Schwartz — Tue, 03 Aug 2010 19:24:26 +0000

I think that people are looking for different things here:

1) get binlogs off the master to save space
2) get binlogs off the master to keep them safe

In my opinion, the best technology for item 2) is probably DRBD at the moment.

By: Mark Callaghan

Mark Callaghan — Tue, 03 Aug 2010 18:20:28 +0000

I would prefer to use network attached or remote storage as the place where binlog archiving is done rather than where the binlogs are written in the first place.

Default InnoDB/MySQL doesn’t report IO latency — I think Percona Server does and I know the Facebook patch also does. I assume that binlog sync to local storage accelerated by HW RAID write cache is much faster than binlog sync to NFS. Given the lack of group commit in InnoDB that can make a huge difference.

By: Mark Callaghan

Mark Callaghan — Tue, 03 Aug 2010 18:18:05 +0000

Yes, this allows for loss of transactions. But losing transactions from the last second might be much better than losing them from the last hour. I need sync replication in MySQL to avoid that or use DRBD to make it less likely. MariaDB is working on that with Galera.

By: Patrick Casey

Patrick Casey — Tue, 03 Aug 2010 18:15:30 +0000

Mark, I think you still have an issue of binlog corruption even if you’ve put the binlogs on an NFS share don’t you? If I lose the network interconnect or the master dies or something else “bad” happens, its not guaranteed that I have a consistent binlog on the NFS server. I might have an incomplete record at the end for example if we were in the middle of a write when the failure occurred.

So if your requirement is *absolutely no* transactions may be lost in a failure, then I don’t think remote mounting the binlogs gets you there, although its probably marginally better than just running a slave.

By: Mark Callaghan

Mark Callaghan — Tue, 03 Aug 2010 18:08:47 +0000

Replication doesn’t archive the master’s binlog. Were that the case, then failing a slave to a new master without losing transactions would be trivial. Today you get to play the game of finding the offset on the new master that corresponds to the offset from the old master. Or you can use global transaction IDs from the Google patch.

Replication is also much more expensive than archiving the binlog. I want to use both but I don’t want to pay for the hardware on a slave when all that I need is a binlog archive solution.

By: Kristian KÃ¶hntopp

Kristian KÃ¶hntopp — Tue, 03 Aug 2010 18:01:33 +0000

Where I work and have been working in the past, NFS fails more often than anything else. Also, the recovery is usually only through a box reboot instead of service restart or even SIGHUP, so this is also the worst possible behavior.

Live binlog shipping exists as well – it is called replication (START SLAVE IO_THREAD, and STOP SLAVE SQL_THREAD, if you are really paranoid; or a time delayed slave, if MySQL were to integrate patches for that into replication after all). But a simple binlog_cycle_command, that would be really easy, and quite useful.

By: Patrick Casey

Patrick Casey — Tue, 03 Aug 2010 17:14:40 +0000

I’ve had NFS mount points lock up on my production servers before, mounting a RedHat NFS point onto a RedHat server.
Triggering factor is usually, but not always, some sort of network interrupt.
I back up onto NFS mount points, but I don’t run anything realtime critical on them.

From my perspective, putting the binlogs on an NFS server triples my failure domain.

I lose the database if:
1) DB server dies
2) NFS mount point dies
3) NFS server dies

I’d rather risk losing a few binlogs than increase my risk of a service interruption. Naturally, everybody’s use case varies