Most applications can be broken down into shards (see my comment #6 for examples), but I don’t dispute this can be difficult in others. The example I often give for an application that won’t shard is IMDB’s database. I don’t think there are many good ways to divide actors up, and the movies they star in.

A small correction to your point about locking: Readers don’t block writers in InnoDB because of MVCC, but MySQL does have locking. Related to your point though is cross-box consistency, and it is an issue. Peter wrote about this in comment #23.

Where a raw disk has a filesystem placed on top of it to aid in organisation of the underlying data,
a database server typically will use table structures on top of a filesystem to further abstract the low level operations of storing information in files into something that can be searched, modified and more easily maintained in a structured form.

Similarly, shards, when implemented in a useful way, are able to abstract a given system in such a way that you’re able to distribute storage across an arbitrary number of machines. In our case, we have shards in different physical locations, where things like replication are completely impractical.

By extension, the reason that sharding isnt really a good idea for most people is the same reason that, for example, if you want to copy your holiday photos onto a usb thumb drive, you dont use a database to do it. In many cases, that extra level of abstraction is completely useless and simply adds complexity.

There are a lot of people who have spent a lot of time researching this area, and, particularly in the web world, it is an invaluable tool for dealing with enormous data sets. I think the notion that it’s ‘hack that ignores 50 years of database theory’ probably just indicates the need for better education and understanding of how it can be used as a tool.

You forgot to mention that if there are many more reads than writes (common case) then running slave, read only databases off a master provides scalability without having to resort to sharding.

Also, Sharding only works if the shards are largely independent, eg. GMail user accounts. But sharding and an integrated system such as ERP is likely to slow it down as the shards need to communicate.

Some databases (Oracle) can horizontally partition a table (and I hope thus a database) automatically based on key values. That is the right approach. Keep the logical/physcial separation. Google style sharding and big table are a hack that ignores 50 years of database theory.

You also forgot to mention that if you take a couple of big tables out of a database, you loose locking and transactions. Not a good option. (Oh, I forgot, MySql does not have locking anyway .)

Sharding is super easy if you know what you’re doing. 1,2 are not an issue at all for me. I can isolate all traffic for super powers users to an in memory DB which will not be overrun if done correctly.

Log Buffer #158

We’re not against sharding. In fact we help a lot of people how to shard properly. The problem is now it is such a buzz word so people with 1GB data set start sharing even if it is never going to grow over 10GB

The bad design is one issue the other however simply working with sharded data. Really it is a lot depended on how tightly coupled is your data – for example hosting million of separate blogs is very easy to shard because there is no interdependencies.

The large data also indeed causes operational concerns – databases in TB range are often not fun in MySQL due to challenges with backups and expecially things as alter table. http://www.mysqlperformanceblog.com/2006/10/08/small-things-are-better/

With backups – the concern is cross box consistency. With single box you can restored backup from yesterday and it will be consistent (even though not up to date) – in sharded envinronment backups will correspond to different point in time and so would not be consistent.

The functional partitioning makes sense under 2 conditions

1) The functional partitions are independent enough, hence you do not need to join data frequently between them at all. Putting different tables on different hosts is not the idea, putting “Forum” on one database host “Wiki” on another and “Bug System” on the third is.

2) The gain you’re looking for is relatively small. It is often easy to find 3 independent functions with one of them responsible for 50% of the load (and hence split giving you double capacity) but getting 10x this way is rarely possible

I have got some interesting numbers back from my analysis though;

My main area of concern was the replacement of data with refreshed information. This involved DELETEing large chunks and INSERTing new data. There is a single field that is primarily used when doing the DELETE, which contains roughly 200 unique entries with a fairly even distribution (eg DELETE FROM `table_name` WHERE `field_name` = 5).

After partitioning the data into 10 chunks based on the HASH of this field, we have experienced a four-fold improvement in the DELETE command, which is now able to DELETE 8.2m records from a 1.5b record table faster than a similar DELETE of 3.6m records from a 500m record table.

Maybe our application just happens to be one of those lucky ones that partitioning _may_ help!

I have yet to get some proper performance numbers on the SELECT side of the data, but I shouldn’t imagine it will be any slower than the current layout, and the data replace section is such a key one, I’m prepared to forego a small performance hit for the increases I’ve already mentioned.

@Dave – When Baron was talking about archiving, he was probably implying mk-archiver – http://www.maatkit.org/doc/mk-archiver.html. It’s an excellent tool if you find you don’t need older records.

I think if the powers that be understood the real cost of MyISAM, that equation would flip on its head.

Maybe you can think about archiving.