Comments on: Is your server’s performance about to degrade?

By: slavik

slavik — Fri, 04 Jun 2010 07:16:20 +0000

maybe this small tool will be useful to detect such micro-freezes, i wrote this to detect on vm/vds when host overloaded
http://sacc.cybersec.ru/files/lagtest.c

By: Jon

Jon — Thu, 27 May 2010 02:41:19 +0000

Actually, in my experience, you shouldn’t touch xfs on top of md or lvm. Only use it on top of hardware RAID. I’ve had multiple cases of worse bugs than the ones mentioned earlier with xfs on md, causing massive data loss. If you are in the position to run an more recent kernel than what comes with Centos 5.x, things might be better….

By: Vojtech Kurka

Vojtech Kurka — Mon, 24 May 2010 21:47:19 +0000

I’ve faced such freezes too. It took me quite a long time to find that DROP/ALTER table on InnoDB tables freezes the whole innodb engine for a few seconds. It’s shorter with small buffer pool, but for 40GB it takes about 2 seconds and the load doesn’t matter.

Just look at bugs http://bugs.mysql.com/bug.php?id=51325 and http://bugs.mysql.com/bug.php?id=51325
The fix is in 5.1.46. However, I haven’t tested it yet.

By: Baron Schwartz

Baron Schwartz — Fri, 21 May 2010 01:36:28 +0000

http://www.mysqlperformanceblog.com/2009/02/05/disaster-lvm-performance-in-snapshot-mode/

I was wrong, it’s not Vadim, it’s Peter.

By: Rob Wultsch

Rob Wultsch — Fri, 21 May 2010 01:18:29 +0000

@Baron Schwartz
Which benchmarks in particular?

By: Baron Schwartz

Baron Schwartz — Thu, 20 May 2010 00:15:38 +0000

I used to be a big fan of LVM, but that changed when a) Vadim’s benchmarks convinced me that it is more costly than I thought, and b) now there’s XtraBackup. I still think it’s nice technology, but I don’t think it’s vital anymore.

I think that bugs from 2007 should not deter you. I cannot recall ever seeing problems with XFS.

By: Mrten

Mrten — Wed, 19 May 2010 12:57:30 +0000

Baron,

I decided against XFS (picking JFS) for our servers when I bumped into these two bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=227331
https://bugzilla.redhat.com/show_bug.cgi?id=240077

I know those are from 2007, but since you’re recommending XFS ‘without hesitation’ I thought I’d ask: Have you ever (more recently) run into problems like those two bugs with XFS/LVM/MD? Or is it that I should just give up on LVM for my db-servers?

thanks for any insight.

By: Morgan Tocker

Morgan Tocker — Wed, 19 May 2010 11:48:29 +0000

I believe it’s been mentioned in a post by Peter before -

One graphed stat I like to read surrounding these sorts of problems is logical rows read/second (one of the last stats in SHOW INNODB STATUS).

If performance is suddenly much worse (but these numbers are remaining about the same), then it may be time investigate how the access pattern changed or if just a little bit of data growth caused far more cache misses.

By: Baron Schwartz

Baron Schwartz — Wed, 19 May 2010 10:19:33 +0000

Flavian, we are happy to help you with consulting, or in forums.percona.com.

Arjen, there’s been a merry fight around the ext* filesystems, which has gained the notorious name “The great fsync() bug” — its behavior is terrible. For the last couple of years I have recommended xfs without hesitation. You will see that’s what Vadim runs all his high-performance benchmarks on too. It’s hard to contemplate a high-performance database server on anything else these days (on Linux, I mean; there are other choices on different OSes).

By: Arjen

Arjen — Wed, 19 May 2010 08:22:53 +0000

We had an entirely different reason to see freezes. We use a ext4 filesystem and have binlog enabled to facilitate replication. Ext4 (at least with default settings) doesn’t flush the data to disk as often as ext3 does, but when you explicitly sync data to disk, it might have to flush up to the entire file-size of data to disk.
Afaik this behaviour has changed a bit over time, but when we were hit by it… it used to flush the entire 1GB binlog from memory to disk at the moment MySQL issued the sync. And that sync is by default only issued inside the binlog-rotation code, within the global logging-lock. I.e. it could stall several seconds because the lock wasn’t released wich was caused by the very long sync-call.

Obviously with faster syncs (i.e. because you sync your binlog more often or with another FS) the freezes will be much shorter, but they still may occur.