Comments on: Recovering Innodb table Corruption

By: peter

peter — Sat, 12 Jul 2008 20:14:13 +0000

Toby,

I ended writing this post well after midnight so I forgot to post the bug. Posted it now - thanks.
http://bugs.mysql.com/bug.php?id=38078

By: Toby

Toby — Sat, 12 Jul 2008 13:51:42 +0000

Did you file a bug on the CHECK TABLE deficiency?

By: Justin Huff

Justin Huff — Tue, 08 Jul 2008 02:53:45 +0000

The trick is to make sure you time the hardware work when it’s REALLY hot in the office. The data center is MUCH cooler:)

By: peter

peter — Tue, 08 Jul 2008 02:42:53 +0000

Oh well,

Then you can try more rounds replacing hardware

By: Justin Huff

Justin Huff — Mon, 07 Jul 2008 23:19:11 +0000

Peter,
Yeah, it’s just a slave for reporting, so it’s not a big deal when it acts up….

By: peter

peter — Mon, 07 Jul 2008 23:16:25 +0000

Justin,

If it is just one box you may just better off moving stuff to other box - in many case simply calling box “buggy” and not using it for critical stuff is more cost effective than spending hundred of hours to figure out what exactly is the issue.

Also I should note if you have problem happening during the week in average you can’t be sure box is “good” by running just 24h worth of tests.

By: Justin Huff

Justin Huff — Mon, 07 Jul 2008 21:56:39 +0000

Baron,
I did swap the card (a 3ware SATA controller), but there is a riser involved. We have ~10 other systems with an identical setup, and this is the only one with issues.

Peter,
I suspected ram at first, so I ran memtest86 for 24hrs with no failures. I think I have one ram permutation left to try before I swap the motherboard.

Sadly, this problem takes over a week to appear in production, so it takes a while to narrow down:(

thanks!

By: peter

peter — Mon, 07 Jul 2008 16:34:34 +0000

Justin,

The memory subsystem (which included mainboard) also can be the problem.
In many cases with random Innodb corruptions doing memory diagnostic with memtest86 or similar tools pointed the problems.

By: Baron Schwartz

Baron Schwartz — Mon, 07 Jul 2008 15:46:48 +0000

How about your RAID controller?

Search Google for what Jeremy Cole has to say about LSI cards and risers — there might be something relevant there. Just a vague memory.

By: Justin Huff

Justin Huff — Mon, 07 Jul 2008 15:41:34 +0000

Funny, we’ve had a DB box acting up (InnoDB complaining about page corruption) for several months. It’s never the same page. The files on disk appear to be OK, it’s like they get corrupted in memory.

I’ve dumped/reloaded the DBs, and swapped nearly everything in the box. The last remaining pieces are the motherboard and the processors.

Yuck:(