Comments on: InnoDB: look after fragmentation

By: Petrik_CZ

Petrik_CZ — Mon, 15 Feb 2010 13:43:51 +0000

After several trials I am doing “defragmentation” of innodb 10M+ rows table by exporting to csv file and then loading to new table. Then I rename them, copy all changes which occured during export/import and delete old table. quite fast compared to other methods.

By: zanzibar

zanzibar — Fri, 22 Jan 2010 07:33:41 +0000

I just ran ‘ALTER TABLE tracker ENGINE=InnoDB’; it takes forever with 20 million rows. I wonder if there’s a better way to defrag on production!

By: Vadim

Vadim — Sat, 21 Nov 2009 19:04:06 +0000

Frank,

I compared different queries as they use different indexes.

Q1 uses primary key and Q2 uses block_id

The idea behind that is if you run query that you think should be executed by PRIMARY KEY you will not
use FORCE KEY (secondary_key) on it. By PRIMARY KEY is usually much faster for InnoDB.

By: Frank

Frank — Mon, 09 Nov 2009 13:29:15 +0000

Vadim
in this case / with that special purpose in mind why do you compare the runtimes of different queries?
Q1=
SELECT count(distinct username) FROM tracker where TIME_ID >= ’2009-07-20 00:00:00′ AND TIME_ID = ’2009-07-20 00:00:00′ AND TIME_ID <= '2009-10-21 00:00:00'

At first glance one would use identical queries and FORCE INDEX usage, wouldn't one?

By: peter

peter — Sat, 07 Nov 2009 18:35:44 +0000

Miguel,

Indeed Innodb clusters data by primary key. This clustering is however per page. For example in case we would use MyISAM we could get a “disk seek” and IO for each row in worse case scenario – in Innodb it does not happen. The data is always going to be clustered on per page basics and hence at least 16K worth of data would be read each time. The number of rows of course depends on row length and page fill factor – in this case we had few hundred rows per page.

Now note in the worse case scenario no read-aheads will trigger and all IO will be done by single thread in 16K blocks. Considering 200 IOS/sec for legacy (non Flash) hard drive you will be looking at about 3MB/sec read speed which is 30-50 times slower than sequential read speed of the same drive.

By: Arjen Lentz

Arjen Lentz — Sat, 07 Nov 2009 00:12:10 +0000

Migael, in InnoDB’s architecture, it works out faster to insert in primary key order. Yes the B+Tree gets rebalanced either way and a B+tree is by definition sorted, but there’s just less work in this particular scenario. That’s why it’s desirable that a dump for InnoDB tables has the rows in PK order.

By: Steven Roussey

Steven Roussey — Fri, 06 Nov 2009 15:56:33 +0000

One thing that was great about mysisam was ALTER TABLE ORDER BY, so you could decide how to order your data…

By: Miguel DeAvila

Miguel DeAvila — Fri, 06 Nov 2009 15:21:26 +0000

Can you say more about the nature of the fragmentation?

Even if the dump occurred in non-pk order, the pk order should have been restored during the import, no?
I thought that the price for the out-of-order restore would be excessive i/o during the restore as the
b-tree is continually re-organized to cope with the out-of-order inserts. After the restore the table must
be in pk order, no?

By: peter

peter — Fri, 06 Nov 2009 01:36:53 +0000

Arjen,

Right. From the size prospective the indexes are about the same. Primary key is even a bit larger as it has more information in it than primary key. However it is still better to scan in primary key order in most cases because it is less fragmented. It is very frequent to see auto increment primary key or other sequential insert patterns.

By: Arjen Lentz

Arjen Lentz — Thu, 05 Nov 2009 23:20:01 +0000

Interesting case.
Technically it was an equal choice in terms of index selection, since the indexed column (block_id) plus the composite primary key covers all the columns. So it was just as valid to pick that index as it was to just scan the primary key. And it would be just as fast.

Of course, when using the output from that query to insert into another table (via a dump or just INSERT … SELECT), and the destination table is InnoDB, then it’s more beneficial to have the select in primary key order. But hey, we all know that unless you specify an ORDER BY, result set order is undefined.
I don’t think this is a optimiser bug, really. If you want the select ordered, use an ORDER BY clause.

And indeed, mysqldump offers the –order-by-primary option, and using it with InnoDB is good. For other engines, the situation can be quite different. For MyISAM, ordering by primary may in fact trigger a filesort (even go to disk) since a full table scan is likely not in primary key order. It’s arbitrary (well, based on insert order + gap filling).