Comments on: Why MySQL could be slow with large tables ?

By: sqi

sqi — Thu, 28 Aug 2008 07:15:03 +0000

By: arpit

arpit — Thu, 28 Aug 2008 07:05:03 +0000

Hello,pls suggest the solution for my problem.
I retrive records from 4 tables which are quite large in size using joins ,but it takes lot of time to execute.How to speed up the same query?

By: trent

trent — Wed, 30 Jul 2008 15:18:46 +0000

So - if you have a table with millions of rows with lots of updates and reads happening, InnoDB would be the way to go from what I read, But want if you want to use mysql ‘full text search’ which can only be used on MyISAM.

Would duplicating data on inserts and updates be an option which would mean having two of the same table, one using InnoDB for main reading purposes and one for MyISAM for searching using Full text search and every time you do an update actually uipdate bith table etc. (forget duplicating data as this is cheap enough) I want speed.

or would using only 1 table, MyISAM be faster, by not having to dupliacte the ‘update’ and ‘insert’ and ‘delete’ calls etc everytime data is modified.

Or is there another was to do this like, run a cron every hour to keep MyISAM table in sinc or somthing, meaning search would be out of date by one hour - which I can live with in my situation.

By: How is MySQL’s join performance these days? | DBMS2 -- DataBase Management System Services

Thu, 10 Jul 2008 22:27:30 +0000

[...] similar to those in a mid-2006 post on MySQL Performancing Blog, which said: One of the reasons elevating this problem in MySQL is lack of advanced join methods at [...]

By: Vladimir

Vladimir — Tue, 17 Jun 2008 12:51:35 +0000

Normally MySQL is rather fast loading data in MyISAM table, but there is exception, which is when it can’t rebuild indexes by sort but builds them
row by row instead. It can be happening due to wrong configuration (ie too small myisam_max_sort_file_size or myisam_max_extra_sort_file_size) or
it could be just lack of optimization, if you’re having large (does not fit in memory) PRIMARY or UNIQUE indexes.

By: YC Wee

YC Wee — Tue, 03 Jun 2008 08:52:04 +0000

Hi guys,
I have recently written an article, “Full Text Partitioning - The Ultimate Scalability”, and my article can be found at the url below:

http://www.addedworth.com/index.php/2008/06/03/full-text-partitioning-the-ultimate-scal

This article describes the steps to take when a database is spilled over to more than a single server. Do come by my site and let me know your opinion.

Thanks,
YC Wee

By: Hugo Rafael

Hugo Rafael — Thu, 01 May 2008 17:06:42 +0000

That seems to be the solution. I had already found that solution on MySql web site, although 5.1 is still not stable according to them, and 5.0 doesn’t support partitioning.
I’ll have to do it like that, and even partitioning over more than one disk in order to distribute disk usage.

Cheers.

By: xeeshaan

xeeshaan — Thu, 01 May 2008 08:45:53 +0000

Best Practice to deal with large DBs is to use a Partitioning Scheme on your DB after doing a thorough analysis of your Queries and your application requirements. Since 5.1 support Data Partitioning, I am using the scheme over a Huge DB of Call Details records which is growing as 20M (approximately 2.5GB in size) records per day and I have found it an appropriate solution to my Large DB issues.

As for Joins, its always best practice not to use joins over Large Tables. Instead use alternate Sub-queries to Joins where possible and with the use of Partitioning make subsets of your data and bring them in Heap-tables rather than storing them on Disk and perform your operations making sub-tasks of Task.

Remember when Anaconda eats a deer it always take time to get it right in itss stomach. So give your Anaconda small pieces of meat than full deer all in once.

By: Hugo Rafael

Hugo Rafael — Mon, 04 Feb 2008 15:23:32 +0000

It does help, cheers.
Dropping the index is out of the question, since dropping them and creating them takes far too much time, being even quicker to just let them be.
I think the answer to this, is just drop the PK and FK’s, and create a normal index with the two main searchable columns. Although this index seams to be a bit slower, I think it might be quicker on large inserts on the table.
I suppose that I’ll have to break the table up, as well, in order to have all the data in smaller tables and smaller indexes.

I was having indexes almost the size of the complete table (+/- 5GB), which made the whole table around 10GB. I have been playing with different indexes and at this time I managed to drop the index’s size to up 1.5GB, which is much more acceptable. Although the selects now take 25% more time to perform, it’s still around 1 second, so it seams quite acceptable to me, since there are more than 100 million records in the table, and if it means that the inserts are faster.

Cheers for your help.

By: Jim

Jim — Mon, 04 Feb 2008 13:14:28 +0000

Sorry for the confusion, but this is what I was talking about also. If you need to search on col1, col2, col3 then create an index(col1,col2,col3). What I was saying is if you have an index(col1), index(col2), and index(col3) and you run a query and want it optimized, the MySQL engine will not combine the second example to give you index(col1,col2,col3) for a more optimized query. It will only pick index(col1) or index(col2), index(col3) or none of the above. It doesn’t take any longer to create any one of these indexes, but I’m sure you know all this. The trick I use, because I have one database that is fairly large with many indexes, is to drop the indexes, then do the bulk up load, and then recreate the indexes.
I hope this helps.