Comments on: Working with large data sets in MySQL http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/ Everything about MySQL Performance Sat, 21 Nov 2009 05:23:57 -0800 http://wordpress.org/?v=2.8.4 hourly 1 By: peter http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/comment-page-1/#comment-146665 peter Wed, 18 Jul 2007 11:22:38 +0000 http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/#comment-146665 Right. For Innodb it is strange order by DESC is much slower. But anyway you should not be using large LIMIT. I think I've blogged about couple of workarounds but they are mostly best for static data Right. For Innodb it is strange order by DESC is much slower.
But anyway you should not be using large LIMIT. I think I’ve blogged about couple of workarounds but they are mostly best for static data

]]> By: Ken http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/comment-page-1/#comment-146654 Ken Wed, 18 Jul 2007 10:45:50 +0000 http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/#comment-146654 I'm using InnoDB tables, so I guess I can't use PACK_KEYS? Is there any other way to go about overcoming this problem? Could you perhaps do a blog on this very topic? I think many sites suffer this kind of dilemma. I’m using InnoDB tables, so I guess I can’t use PACK_KEYS?

Is there any other way to go about overcoming this problem?

Could you perhaps do a blog on this very topic? I think many sites suffer this kind of dilemma.

]]> By: peter http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/comment-page-1/#comment-146595 peter Wed, 18 Jul 2007 07:42:39 +0000 http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/#comment-146595 Right. You can't use large limits because skipping rows becomes expensive. Also DESC can be slower for MyISAM Tables with key compression you can set PACK_KEYS=0 for this table and check if it helps. Anyway this becomes rather offtopic for this post :) Right. You can’t use large limits because skipping rows becomes expensive.
Also DESC can be slower for MyISAM Tables with key compression you can set PACK_KEYS=0 for this table and check if it helps.

Anyway this becomes rather offtopic for this post :)

]]> By: Ken http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/comment-page-1/#comment-146510 Ken Wed, 18 Jul 2007 03:19:31 +0000 http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/#comment-146510 Yes I do have an index there. The problem is when you start asking for the last few pages, like this: SELECT u.id from sys_user u WHERE u.account_status='Active' ORDER BY u.last_updated desc limit 188290,10; The problem is with the "desc limit 188290,10" it seems, because if I remove the "desc" or change 188290 to 1, the query time becomes much shorter. Yes I do have an index there. The problem is when you start asking for the last few pages, like this:

SELECT u.id from sys_user u WHERE u.account_status=’Active’ ORDER BY u.last_updated desc limit 188290,10;

The problem is with the “desc limit 188290,10″ it seems, because if I remove the “desc” or change 188290 to 1, the query time becomes much shorter.

]]> By: peter http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/comment-page-1/#comment-146249 peter Tue, 17 Jul 2007 16:11:16 +0000 http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/#comment-146249 Ken, Do you have index on last_updated ? Not to mention you can cheat and cache pages for a few seconds. Ken,

Do you have index on last_updated ?

Not to mention you can cheat and cache pages for a few seconds.

]]> By: Ken http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/comment-page-1/#comment-146237 Ken Tue, 17 Jul 2007 15:16:52 +0000 http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/#comment-146237 What about cases where I'm returning a long list of users, sorted by their last login times? I'm using paging (limit) in my mysql, but I've noticed that when the result set is huge, this becomes a nightmare. Without the sorting things are fast, but with sorting, it slows down to a crawl, especially at the last pages. What do you suggest here? I've seen some sites like Myspace, that just return the first 3,000 results (not sure what they've done exactly). What about cases where I’m returning a long list of users, sorted by their last login times? I’m using paging (limit) in my mysql, but I’ve noticed that when the result set is huge, this becomes a nightmare. Without the sorting things are fast, but with sorting, it slows down to a crawl, especially at the last pages.

What do you suggest here? I’ve seen some sites like Myspace, that just return the first 3,000 results (not sure what they’ve done exactly).

]]> By: peter http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/comment-page-1/#comment-145046 peter Fri, 13 Jul 2007 18:31:18 +0000 http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/#comment-145046 Yes this should work OK. The Joins are the main showstopper for using a lot of tables. Yes this should work OK.

The Joins are the main showstopper for using a lot of tables.

]]> By: howie http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/comment-page-1/#comment-145043 howie Fri, 13 Jul 2007 18:26:24 +0000 http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/#comment-145043 I've been struggling with this problem of large data sets (well, large for me). I'm afraid of switching to 5.1 and using partitioning. Do you basically roll your own partitioning? Do you just choose something to partition on and then create a separate table for it? For example, a big table called invoices might be split into invoices_2007, invoices_2006, etc. I was thinking of doing this for one large table. I basically do two types of queries on the table, so I think I might need to mirror the data and partition on two separate fields. Using the invoices example, I would partition by year, and have another set of tables partitioned by vendor name (invoices_vendor_a, invoices_vendor_b, etc). Does this sounds like a reasonable approach? In my case, I won't have to join across partitions, so it seems like it should work. I’ve been struggling with this problem of large data sets (well, large for me). I’m afraid of switching to 5.1 and using partitioning.

Do you basically roll your own partitioning? Do you just choose something to partition on and then create a separate table for it? For example, a big table called invoices might be split into invoices_2007, invoices_2006, etc.

I was thinking of doing this for one large table. I basically do two types of queries on the table, so I think I might need to mirror the data and partition on two separate fields. Using the invoices example, I would partition by year, and have another set of tables partitioned by vendor name (invoices_vendor_a, invoices_vendor_b, etc).

Does this sounds like a reasonable approach? In my case, I won’t have to join across partitions, so it seems like it should work.

]]>