As described in this blog post, a table scan with order by/limit does a very fast sort if you want to get the top-most rows .. but I find it useless if you want to use it to create a full ordered insert (for example) for innodb.
My large table spends hours ’sorting result’, even though I can get top 20 results within 15 minutes. I wonder if there are some short-cuts taken by mysql where limit is used. Perhaps a table-scan, with a running best-of list, for example. That would not then be a sort, would it?

In order to do an ordered bulk insert to innodb by PK I am resorting to external sort of a file with unix sort then feeding it with batch inserts, rather than feeding it to myisam then converting, or feeding to myisam then sort .. order by .. insert, or feeding un-ordered PK to innodb directly then doing alter table engine=innodb to “defragment” PK.

Am I on the right track?

a) Add an index on column to be sorted and select by that index
b) Dump the data to flat file and sort using unix sort then re-import
c) Select using table-scan and sort as per your example
d) alter to innodb with primary key (which is the objective anyway)

I’m currently doing (a) but I’m wondering if this is a big waste of time.

Indeed it can affect things. And there are more variables like table structure of course – I have not tested these yet.

I wonder how the size of this buffer would affect real loaded systems?

Obviously on many real systems disk IO comes into the equation and then disk seek time can start playing a big factor.

I would be interested to see if the results change in this kind of situation.

Kind Regards,
Lachlan

Whitesites, larger buffers often still do help performance on mixed query loads but what this does show is that if you can have the default low and increase it only in the sessions that benefit from larger sizes you can get the best of both worlds. As you optimise queries and indexes you can also find that the optimal value for a particular workload changes.