Using direct data load will be faster. How much faster depends on a a lot of factors – data types and size, amount of indexes etc.

I am sorry that I spoke too soon. I tried it again, and it seemed working. thanks Peter!

Jason

Thanks do much for the prompt reply!

I tried the table partitioning, but it did not seem to work. I might not partition on the right values. but again, how do I find out a table is partitioned? “desc table_name” dose no show any information on the partitioning.

thanks,

Jason

The slowdown happens because BTREE (which tends to get a lot of random IO) becomes too large to fit in cache.
The solution is simple – keep BTREE small enough so it does not happen.

In MySQL 5.1 you can use partitions to use just this, in MySQL 5.0 and below you can just use multiple tables.

Thanks for this wonder blog on the load performance. It is so timely. I was trying to evaluate MySql performance to see if it could fulfill our requirement. I found the data insertioin speed slows down drammatically (about 300 times slower) when the table size reaches to a certain number, just like you mentioned in this blog. I tuned on the paramerters like innodb_buffer_pool_size, large innodb_log_file_size, and a few other ones to the max values my machine can take. It does increase the speed and also push the threshold of the break point to a much larger table size. but I can not avoid the break down of the insertion speed completely. It will happen sooner or later. My question is that is this type of insertion speed drammtic slow down scenario can never be avoided no matter how large the CPU and RAM are? The result of larger CPU and RAM can only push the break down further on a larger table size, but can completely elimiate it?

I did this test on MySql 5.0 and my laptop has 2.6 GHZ CPU and 2GB Memory. I found the insertion speed slow down at about 1GB table size. I use JDBC to insert 50,000 records with one insertion. the insertion time is 3 second. when it reach Number 100 insertion, the insertion time eventually reaches 900 seconds and still increase.

Thank you very much with your earliest help!

Jason