Comments on: How many partitions can you have ?

By: pcrews

pcrews — Tue, 03 Jan 2012 23:17:56 +0000

http://bugs.mysql.com/bug.php?id=37252

By: oliver

oliver — Wed, 03 Feb 2010 18:50:07 +0000

what you mean with having multible Files running partitioning? even with InnoDB? how?

regards

By: Tonico

Tonico — Sat, 02 Jan 2010 00:41:07 +0000

Do not jump into wrong conclusions. On a production system, if you are partitioning, one of the benefits of having partitioning is to point each partition to a separate volume with eventually independent disk controllers. The level of parallel disk access would drastically benefit performance. If you have multiple partitions on the same disk/disk controller you are getting high I/O waits.

By: Mikael

Mikael — Sat, 12 Dec 2009 12:23:06 +0000

See BUG#48846 for some interesting development in this area

By: peter

peter — Tue, 08 Dec 2009 03:34:47 +0000

Harrison,

I just checked I do not see any difference with LOCK TABLES and without LOCK TABLE – I’m doing inserts by 1000 rows in the batch. As I mentioned I tested and I still see overhead with even more bulky statements so it looks like there is some non negligable overhead per row (not only per statement)

By: peter

peter — Tue, 08 Dec 2009 03:25:39 +0000

Harrison,

I kind of checked that by using inserts with different level of “bulkiness” Table lock should be happening only once per statement hence overhead should be lower if you have more bulky statements which does not seems to be the case in this case. Also it does not explain increase cost of update.

By: Britt

Britt — Mon, 07 Dec 2009 18:02:54 +0000

Say I have a table with 100 partitions, partitioned by some column id. If I look up rows from this partitioned table using an IN clause with 200 ids, it seems like it first figures out which partitions it needs to check, then looks in each partition for all 200 ids, causing a lot of index reads and slowness. When I split the large select query manually to do one query per partition, I saw a big speedup. Maybe something similar is happening with your insert…update?

By: Harrison Fisk

Harrison Fisk — Mon, 07 Dec 2009 17:29:54 +0000

Have you seen http://bugs.mysql.com/bug.php?id=37252 ? This implies it is the locking that causes the slowdown (even for InnoDB/Falcon). Can you try your test with locking the table first, doing the INSERT ON DUPLICATE KEY UPDATE statements, and then unlock?

By: peter

peter — Mon, 07 Dec 2009 16:47:56 +0000

Istvan,

What you’re saying applies to not partitioned key. If you lookup by the key which you partition on only one partition should be looked up. If it is other key it has to check all partition which is indeed can be significant overhead. I planned running some tests on it later.

By: Istvan Podor

Istvan Podor — Mon, 07 Dec 2009 06:19:25 +0000

Hey Peter,

In my experience it’s not just about writes. If you have 1k table and with a lots of data (50k rows) per each table and you try to query against an indexed data, mysql will go trough each index and if the index’s size is around a few megabytes for each table, this can dramatically slow down the usually fast unique/primary key look ups (as all partition(table) got its own indexes right?). It was the same for me with myisam-merge (what we could think about like an earlier partitioned table). And of course, we can’t forget about if you have a lots of queries where it have to go trough all of the indexes for each partition, that will screw up your index caches and may eat up a lots of memory and decrease your cache hits.

But, benchmark speak