I recommend a consistency check every month.
RAID is absolutely NO substitute for a backup. I think of it as a convenience; that is all.
Not that it matters, but I do work for dell as L2 support.
Backups people!

That way if a bad block is encountered, the data could be resolved from the redundancy.
Also, before marking a drive with a bad block as failed, it should try to write the data back to the disk with the bad block.
This should cause the disk to try and remap the bad block.

RAID 6 should be capable of this already, but i’m very doubtful that many controllers handle bad blocks in this way.

Nice to find you linking to this document. I was sending many customers to read it.

I also generally recommend using RAID10 but I avoid saying it is always must. Each case is unique in practice.

Cost factor would often be the reason but not only that. In some tests I’ve done I found RAID5 to perform _faster_ or close to RAID10 on PowerEdge 2850 for example. You can see benchmark data in “Performance Landscape” presentation from this site. Yes this does not make much sense but I’m not only the person to observe this behavior. Here are some benchmarks for SQL Server
http://www.developersdex.com/sql/message.asp?p=580&r=4986921

My felling is Dell or LSI just spend more time optimizing RAID5 or there are some serve performance bugs in RAID10 implementation as I do not see any physical reasons why this would be happening.

Other reason could be - certain hardware might not have proper RAID10, sometime what is named RAID10 might be implemented as contatenated RAID1 (especially in some older models) - this would have pretty bad performance in many cases.

Besides hardware limitations/bugs I can see RAID5 used in replicated envinronment with low volume of writes. It gives certain sequrity for the slaves so you do not have to reclone them whenever any drive fails also this means you can promote slave to the
master without running master on insecure storage.

You’re right with RAID10 the probability of loosing data is lower, even though it also can fail loosing second hard drive, if they come from the same stripe. Meaning you should still want to use hot spare to minimize such window.

Hot spare would not have helped in this case however - first drive did not fail but was manually replaced so it just started to resync as hot spare would. It however failed due to bad block on the second drive.

In general the point is there is data of different level of importance and there are different decicions comming from it. Of course it is good to have data on RAID10 with spare disk, have reduntant servers and of course backup which you can do point in time recovery from. It is however not always the case in real systems.

Yes of course you should have multiple servers, that is much better for high availability cases especially as RAID is one of the components which fails. However building “Google Style” system build from a lot of crapy hardware might not always be best solution. There are a lot of things to consider - for example Power requirements which is often main cost factor in Colocation envinronments, maintainance - recovering “inexpensive” box might be pretty expensive, especially if it is installed in remote location, not to mention various wierd problems you might need to be ready for - database corruption due to bad memory/cooling which sometimes could be replicated, so even replication might not save you.

So my choice is normally to have decent boxes for MySQL servers. Pretty commodify ones, no high end but reliable. Especially for small/medium company sizes when there is no time to implement management infrastructure which would allow to replace broken servers cheap and transparently.

Thanks for you hints. Yes running consistency check on regular basics is surely good idea. What is interesting in Dell/LSI docs “Patrol Read” is positioned as lower overhead alternative to consistency check. What I’ve found out however is - it does not really catches errors well enough (as in this case) plus it has some strange performance problems - in certain cases I’ve seen it slowing down array to probably 20% of its capacity for 20-30min. Could be bug but Dell just told to disable Patrol read.

It’s an association of knowledgeable sysadmin who won’t ever use RAID5 (or 3 or 4) anymore on production system…

RAID5 alone is dangerous, you can mitigate the risk with a hotspare drive, but frankly, RAID10 is really better (for a lots of good reasons: http://www.miracleas.com/BAARF/RAID5_versus_RAID10.txt)
Disks are cheaps nowadays…

At the very least, you should consider setting aside one disk as a hotspare to reduce the time window of having a second disk fail while the RAID5 array is degraded.

For the price of a RAID card you can buy another cheap server. If you can load balance SELECTs across the boxes and have few writes you can get RAID performance and reliability with numerous cheap MySQL boxes.

Commodity hardware is cheap and modern disks are pretty damn fast if you don’t need ONE box to exec all your queries.