Comments on: Working with large data sets in MySQL http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/ Percona's MySQL & InnoDB performance and scalability blog Sat, 11 Feb 2012 16:45:54 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: Macoway Advertising http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/comment-page-1/#comment-725707 Macoway Advertising Tue, 16 Feb 2010 08:53:05 +0000 http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/#comment-725707 A big internet Search Archive is run and kept on the first full sponsored / collective knowledge / search engine of the world. This System is build from scratch, is 100 % original and is builded in php/mysql .... but it will grow and things will get complicated maybe. Please Visit : www.Macoway.com / they are looking for good mysql database design to cover future development and handle large data sets , as they will probably deal with big databases. A big internet Search Archive is run and kept on the first full sponsored / collective knowledge / search engine of the world. This System is build from scratch, is 100 % original and is builded in php/mysql …. but it will grow and things will get complicated maybe. Please Visit : http://www.Macoway.com / they are looking for good mysql database design to cover future development and handle large data sets , as they will probably deal with big databases.

]]> By: GAC http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/comment-page-1/#comment-715277 GAC Thu, 28 Jan 2010 09:25:44 +0000 http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/#comment-715277 I have a problem, I am using MyISAM engine and I am trying just to fill first the database with data. My data will fit in around 10 tables of 90Gb each one, After it is filled I only need to extract data, well also may be do some statistics. But no transaction, no updates, no inserts anymore. So, my problem is simply, how can I fill this information fast. I used one primary key with three index, and one index separated, this because after it is filled in, I should be able to make fast queries. So the point is, should I remove the index to fill in the data base, would it be faster if I use a different engine? I already played some how with the my.ini but still not improvement in the uploading performance. I take 1 hour for each 200 MB, so you can Imagine the time that it will take me to fill in all. I am uploading chunks of data using load file into, and it is very slow. I am also thinking in partition the table, but I read in other blocks that it might not be the best? althought I know there is no magic solution, I hope someone can give me some advise to dont loose too much time testing all possible combination of solutions. Thanks in advance I have a problem, I am using MyISAM engine and I am trying just to fill first the database with data. My data will fit in around 10 tables of 90Gb each one, After it is filled I only need to extract data, well also may be do some statistics. But no transaction, no updates, no inserts anymore. So, my problem is simply, how can I fill this information fast. I used one primary key with three index, and one index separated, this because after it is filled in, I should be able to make fast queries. So the point is, should I remove the index to fill in the data base, would it be faster if I use a different engine? I already played some how with the my.ini but still not improvement in the uploading performance. I take 1 hour for each 200 MB, so you can Imagine the time that it will take me to fill in all. I am uploading chunks of data using load file into, and it is very slow. I am also thinking in partition the table, but I read in other blocks that it might not be the best? althought I know there is no magic solution, I hope someone can give me some advise to dont loose too much time testing all possible combination of solutions.

Thanks in advance

]]> By: peter http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/comment-page-1/#comment-146665 peter Wed, 18 Jul 2007 11:22:38 +0000 http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/#comment-146665 Right. For Innodb it is strange order by DESC is much slower. But anyway you should not be using large LIMIT. I think I've blogged about couple of workarounds but they are mostly best for static data Right. For Innodb it is strange order by DESC is much slower.
But anyway you should not be using large LIMIT. I think I’ve blogged about couple of workarounds but they are mostly best for static data

]]> By: Ken http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/comment-page-1/#comment-146654 Ken Wed, 18 Jul 2007 10:45:50 +0000 http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/#comment-146654 I'm using InnoDB tables, so I guess I can't use PACK_KEYS? Is there any other way to go about overcoming this problem? Could you perhaps do a blog on this very topic? I think many sites suffer this kind of dilemma. I’m using InnoDB tables, so I guess I can’t use PACK_KEYS?

Is there any other way to go about overcoming this problem?

Could you perhaps do a blog on this very topic? I think many sites suffer this kind of dilemma.

]]> By: peter http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/comment-page-1/#comment-146595 peter Wed, 18 Jul 2007 07:42:39 +0000 http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/#comment-146595 Right. You can't use large limits because skipping rows becomes expensive. Also DESC can be slower for MyISAM Tables with key compression you can set PACK_KEYS=0 for this table and check if it helps. Anyway this becomes rather offtopic for this post :) Right. You can’t use large limits because skipping rows becomes expensive.
Also DESC can be slower for MyISAM Tables with key compression you can set PACK_KEYS=0 for this table and check if it helps.

Anyway this becomes rather offtopic for this post :)

]]> By: Ken http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/comment-page-1/#comment-146510 Ken Wed, 18 Jul 2007 03:19:31 +0000 http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/#comment-146510 Yes I do have an index there. The problem is when you start asking for the last few pages, like this: SELECT u.id from sys_user u WHERE u.account_status='Active' ORDER BY u.last_updated desc limit 188290,10; The problem is with the "desc limit 188290,10" it seems, because if I remove the "desc" or change 188290 to 1, the query time becomes much shorter. Yes I do have an index there. The problem is when you start asking for the last few pages, like this:

SELECT u.id from sys_user u WHERE u.account_status=’Active’ ORDER BY u.last_updated desc limit 188290,10;

The problem is with the “desc limit 188290,10″ it seems, because if I remove the “desc” or change 188290 to 1, the query time becomes much shorter.

]]> By: peter http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/comment-page-1/#comment-146249 peter Tue, 17 Jul 2007 16:11:16 +0000 http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/#comment-146249 Ken, Do you have index on last_updated ? Not to mention you can cheat and cache pages for a few seconds. Ken,

Do you have index on last_updated ?

Not to mention you can cheat and cache pages for a few seconds.

]]> By: Ken http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/comment-page-1/#comment-146237 Ken Tue, 17 Jul 2007 15:16:52 +0000 http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/#comment-146237 What about cases where I'm returning a long list of users, sorted by their last login times? I'm using paging (limit) in my mysql, but I've noticed that when the result set is huge, this becomes a nightmare. Without the sorting things are fast, but with sorting, it slows down to a crawl, especially at the last pages. What do you suggest here? I've seen some sites like Myspace, that just return the first 3,000 results (not sure what they've done exactly). What about cases where I’m returning a long list of users, sorted by their last login times? I’m using paging (limit) in my mysql, but I’ve noticed that when the result set is huge, this becomes a nightmare. Without the sorting things are fast, but with sorting, it slows down to a crawl, especially at the last pages.

What do you suggest here? I’ve seen some sites like Myspace, that just return the first 3,000 results (not sure what they’ve done exactly).

]]> By: peter http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/comment-page-1/#comment-145046 peter Fri, 13 Jul 2007 18:31:18 +0000 http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/#comment-145046 Yes this should work OK. The Joins are the main showstopper for using a lot of tables. Yes this should work OK.

The Joins are the main showstopper for using a lot of tables.

]]> By: howie http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/comment-page-1/#comment-145043 howie Fri, 13 Jul 2007 18:26:24 +0000 http://www.mysqlperformanceblog.com/2007/07/05/working-with-large-data-sets-in-mysql/#comment-145043 I've been struggling with this problem of large data sets (well, large for me). I'm afraid of switching to 5.1 and using partitioning. Do you basically roll your own partitioning? Do you just choose something to partition on and then create a separate table for it? For example, a big table called invoices might be split into invoices_2007, invoices_2006, etc. I was thinking of doing this for one large table. I basically do two types of queries on the table, so I think I might need to mirror the data and partition on two separate fields. Using the invoices example, I would partition by year, and have another set of tables partitioned by vendor name (invoices_vendor_a, invoices_vendor_b, etc). Does this sounds like a reasonable approach? In my case, I won't have to join across partitions, so it seems like it should work. I’ve been struggling with this problem of large data sets (well, large for me). I’m afraid of switching to 5.1 and using partitioning.

Do you basically roll your own partitioning? Do you just choose something to partition on and then create a separate table for it? For example, a big table called invoices might be split into invoices_2007, invoices_2006, etc.

I was thinking of doing this for one large table. I basically do two types of queries on the table, so I think I might need to mirror the data and partition on two separate fields. Using the invoices example, I would partition by year, and have another set of tables partitioned by vendor name (invoices_vendor_a, invoices_vendor_b, etc).

Does this sounds like a reasonable approach? In my case, I won’t have to join across partitions, so it seems like it should work.

]]>