Comments on: How fast can you sort data with MySQL ? http://www.mysqlperformanceblog.com/2007/08/18/how-fast-can-you-sort-data-with-mysql/ Everything about MySQL Performance Tue, 14 Oct 2008 07:04:45 +0000 http://wordpress.org/?v=2.5.1 By: Project 2061 Techlog » Optimizing MySQL Server Runtime Parameters http://www.mysqlperformanceblog.com/2007/08/18/how-fast-can-you-sort-data-with-mysql/#comment-296421 Project 2061 Techlog » Optimizing MySQL Server Runtime Parameters Wed, 07 May 2008 18:55:24 +0000 http://www.mysqlperformanceblog.com/2007/08/18/how-fast-can-you-sort-data-with-mysql/#comment-296421 [...] MySQL Performance Blog: How fast can you sort data with MySQL? [...] [...] MySQL Performance Blog: How fast can you sort data with MySQL? [...]

]]> By: Justin http://www.mysqlperformanceblog.com/2007/08/18/how-fast-can-you-sort-data-with-mysql/#comment-171970 Justin Tue, 25 Sep 2007 05:31:49 +0000 http://www.mysqlperformanceblog.com/2007/08/18/how-fast-can-you-sort-data-with-mysql/#comment-171970 Further to my last question.. As described in this blog post, a table scan with order by/limit does a very fast sort if you want to get the top-most rows .. but I find it useless if you want to use it to create a full ordered insert (for example) for innodb. My large table spends hours 'sorting result', even though I can get top 20 results within 15 minutes. I wonder if there are some short-cuts taken by mysql where limit is used. Perhaps a table-scan, with a running best-of list, for example. That would not then be a sort, would it? In order to do an ordered bulk insert to innodb by PK I am resorting to external sort of a file with unix sort then feeding it with batch inserts, rather than feeding it to myisam then converting, or feeding to myisam then sort .. order by .. insert, or feeding un-ordered PK to innodb directly then doing alter table engine=innodb to "defragment" PK. Am I on the right track? Further to my last question..

As described in this blog post, a table scan with order by/limit does a very fast sort if you want to get the top-most rows .. but I find it useless if you want to use it to create a full ordered insert (for example) for innodb.
My large table spends hours ’sorting result’, even though I can get top 20 results within 15 minutes. I wonder if there are some short-cuts taken by mysql where limit is used. Perhaps a table-scan, with a running best-of list, for example. That would not then be a sort, would it?

In order to do an ordered bulk insert to innodb by PK I am resorting to external sort of a file with unix sort then feeding it with batch inserts, rather than feeding it to myisam then converting, or feeding to myisam then sort .. order by .. insert, or feeding un-ordered PK to innodb directly then doing alter table engine=innodb to “defragment” PK.

Am I on the right track?

]]> By: Justin http://www.mysqlperformanceblog.com/2007/08/18/how-fast-can-you-sort-data-with-mysql/#comment-171897 Justin Tue, 25 Sep 2007 02:04:36 +0000 http://www.mysqlperformanceblog.com/2007/08/18/how-fast-can-you-sort-data-with-mysql/#comment-171897 On this subject.. if you have a huge un-ordered myisam table and wish to get it into sort order before dumping it to INNODB, which is going to be faster? a) Add an index on column to be sorted and select by that index b) Dump the data to flat file and sort using unix sort then re-import c) Select using table-scan and sort as per your example d) alter to innodb with primary key (which is the objective anyway) I'm currently doing (a) but I'm wondering if this is a big waste of time. On this subject.. if you have a huge un-ordered myisam table and wish to get it into sort order before dumping it to INNODB, which is going to be faster?

a) Add an index on column to be sorted and select by that index
b) Dump the data to flat file and sort using unix sort then re-import
c) Select using table-scan and sort as per your example
d) alter to innodb with primary key (which is the objective anyway)

I’m currently doing (a) but I’m wondering if this is a big waste of time.

]]> By: peter http://www.mysqlperformanceblog.com/2007/08/18/how-fast-can-you-sort-data-with-mysql/#comment-169796 peter Wed, 19 Sep 2007 10:20:31 +0000 http://www.mysqlperformanceblog.com/2007/08/18/how-fast-can-you-sort-data-with-mysql/#comment-169796 Lachlan, Indeed it can affect things. And there are more variables like table structure of course - I have not tested these yet. Lachlan,

Indeed it can affect things. And there are more variables like table structure of course - I have not tested these yet.

]]> By: Lachlan Mulcahy http://www.mysqlperformanceblog.com/2007/08/18/how-fast-can-you-sort-data-with-mysql/#comment-169703 Lachlan Mulcahy Wed, 19 Sep 2007 05:40:04 +0000 http://www.mysqlperformanceblog.com/2007/08/18/how-fast-can-you-sort-data-with-mysql/#comment-169703 Hi Peter, I wonder how the size of this buffer would affect real loaded systems? Obviously on many real systems disk IO comes into the equation and then disk seek time can start playing a big factor. I would be interested to see if the results change in this kind of situation. Kind Regards, Lachlan Hi Peter,

I wonder how the size of this buffer would affect real loaded systems?

Obviously on many real systems disk IO comes into the equation and then disk seek time can start playing a big factor.

I would be interested to see if the results change in this kind of situation.

Kind Regards,
Lachlan

]]> By: MySQL Performance Blog » MySQL: what read_buffer_size value is optimal ? http://www.mysqlperformanceblog.com/2007/08/18/how-fast-can-you-sort-data-with-mysql/#comment-169054 MySQL Performance Blog » MySQL: what read_buffer_size value is optimal ? Mon, 17 Sep 2007 14:41:51 +0000 http://www.mysqlperformanceblog.com/2007/08/18/how-fast-can-you-sort-data-with-mysql/#comment-169054 [...] benchmarks and performance research. I just recently wrote about rather surprising results with sort performance and today I've discovered even read_buffer_size selection may be less than [...] [...] benchmarks and performance research. I just recently wrote about rather surprising results with sort performance and today I’ve discovered even read_buffer_size selection may be less than [...]

]]> By: James Day http://www.mysqlperformanceblog.com/2007/08/18/how-fast-can-you-sort-data-with-mysql/#comment-165000 James Day Sat, 08 Sep 2007 00:27:10 +0000 http://www.mysqlperformanceblog.com/2007/08/18/how-fast-can-you-sort-data-with-mysql/#comment-165000 Scott, an index that can satisfy both the WHERE and ORDER BY clauses can be expected to eliminate the need to sort. Whitesites, larger buffers often still do help performance on mixed query loads but what this does show is that if you can have the default low and increase it only in the sessions that benefit from larger sizes you can get the best of both worlds. As you optimise queries and indexes you can also find that the optimal value for a particular workload changes. Scott, an index that can satisfy both the WHERE and ORDER BY clauses can be expected to eliminate the need to sort.

Whitesites, larger buffers often still do help performance on mixed query loads but what this does show is that if you can have the default low and increase it only in the sessions that benefit from larger sizes you can get the best of both worlds. As you optimise queries and indexes you can also find that the optimal value for a particular workload changes.

]]> By: whitesites http://www.mysqlperformanceblog.com/2007/08/18/how-fast-can-you-sort-data-with-mysql/#comment-163909 whitesites Wed, 05 Sep 2007 19:38:11 +0000 http://www.mysqlperformanceblog.com/2007/08/18/how-fast-can-you-sort-data-with-mysql/#comment-163909 Awesome article. Its interesting to find that giving mysql more memory does not help performance. This is totally opposite to what many believe. I am sure this is a result of the amount of System Cache available to mysql. Makes alot of sense considering intel has been moving away from high CPU speeds to beefed up 4, 6 and 8 MB cache with multiple cores. I would be curious to see what the speed improvement is when using one of these beefed up chips. Awesome article. Its interesting to find that giving mysql more memory does not help performance. This is totally opposite to what many believe. I am sure this is a result of the amount of System Cache available to mysql. Makes alot of sense considering intel has been moving away from high CPU speeds to beefed up 4, 6 and 8 MB cache with multiple cores. I would be curious to see what the speed improvement is when using one of these beefed up chips.

]]> By: Jay Pipes http://www.mysqlperformanceblog.com/2007/08/18/how-fast-can-you-sort-data-with-mysql/#comment-162853 Jay Pipes Mon, 03 Sep 2007 17:25:29 +0000 http://www.mysqlperformanceblog.com/2007/08/18/how-fast-can-you-sort-data-with-mysql/#comment-162853 Apachez, We had a very good meeting in New York and you'll be hearing more about this in the coming weeks on my blog and Giuseppe's. Stay tuned. :) -jay Apachez,

We had a very good meeting in New York and you’ll be hearing more about this in the coming weeks on my blog and Giuseppe’s. Stay tuned. :)

-jay

]]> By: peter http://www.mysqlperformanceblog.com/2007/08/18/how-fast-can-you-sort-data-with-mysql/#comment-162197 peter Sun, 02 Sep 2007 17:09:54 +0000 http://www.mysqlperformanceblog.com/2007/08/18/how-fast-can-you-sort-data-with-mysql/#comment-162197 Yes... We should create commented my.cnf examples sometime. Yes… We should create commented my.cnf examples sometime.

]]>