Comments on: How to load large files safely into InnoDB with LOAD DATA INFILE http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/ Everything about MySQL Performance Tue, 02 Dec 2008 19:21:30 +0000 http://wordpress.org/?v=2.5.1 By: Baron Schwartz http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/#comment-359384 Baron Schwartz Fri, 03 Oct 2008 13:28:06 +0000 http://www.mysqlperformanceblog.com/?p=424#comment-359384 Only MyISAM supports full-text indexing in MySQL. If you have a lot of content to index (bigger than your available memory) and you need high performance, you probably need an external solution such as Sphinx or Lucene. Sphinx has a storage engine for easy integration with MySQL. Only MyISAM supports full-text indexing in MySQL. If you have a lot of content to index (bigger than your available memory) and you need high performance, you probably need an external solution such as Sphinx or Lucene. Sphinx has a storage engine for easy integration with MySQL.

]]> By: LeRoy Grubbs http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/#comment-359223 LeRoy Grubbs Thu, 02 Oct 2008 21:24:27 +0000 http://www.mysqlperformanceblog.com/?p=424#comment-359223 What's the best way to load lots of large and small files for full text indexing? Which database engine is best suited for FTI? of large files? What’s the best way to load lots of large and small files for full text indexing? Which database engine is best suited for FTI? of large files?

]]> By: Baron Schwartz http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/#comment-336167 Baron Schwartz Mon, 28 Jul 2008 13:13:57 +0000 http://www.mysqlperformanceblog.com/?p=424#comment-336167 Hi Kye, Please use the Contact Us form on our website http://www.percona.com, as this goes into our ticketing system. Hi Kye,

Please use the Contact Us form on our website http://www.percona.com, as this goes into our ticketing system.

]]> By: Kye Lee http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/#comment-336165 Kye Lee Mon, 28 Jul 2008 13:11:59 +0000 http://www.mysqlperformanceblog.com/?p=424#comment-336165 Please send me the private email with contact info. Thanks Kye Lee Please send me the private email with contact info.

Thanks
Kye Lee

]]> By: Baron Schwartz http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/#comment-334731 Baron Schwartz Fri, 25 Jul 2008 23:29:35 +0000 http://www.mysqlperformanceblog.com/?p=424#comment-334731 I would suggest breaking it into smaller pieces, but it sounds like you have other problems and need a completely different approach -- perhaps the problem is that you even need these bulk loads. Beyond that, I won't say; this is what we do for a living :-) I would suggest breaking it into smaller pieces, but it sounds like you have other problems and need a completely different approach — perhaps the problem is that you even need these bulk loads. Beyond that, I won’t say; this is what we do for a living :-)

]]> By: Kye Lee http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/#comment-334538 Kye Lee Fri, 25 Jul 2008 15:48:23 +0000 http://www.mysqlperformanceblog.com/?p=424#comment-334538 I found your article and thought it is very interesting – Thanks. As result of your test, do you have recommendation or method of efficiently loading very large file? BTW – I am NOT heavy DB programmer and don’t know much about DB. If you don’t mine I would like to seek your advice and help. I have about 120,000 rows – rec size 130 bytes with about 13 fields (Avg 15 GB), which need to be inserted into InnoDB table every min. I am using LOAD command to accomplish this but in some occasion , the LOAD command takes longer than 1 min. When this happened, the following LOAD file get bigger and bigger and eventually, I get DB gone away error and the program abort. Any suggestions. Kye Lee I found your article and thought it is very interesting – Thanks.
As result of your test, do you have recommendation or method of efficiently loading very large file?

BTW – I am NOT heavy DB programmer and don’t know much about DB. If you don’t mine I would like to seek your advice and help.

I have about 120,000 rows – rec size 130 bytes with about 13 fields (Avg 15 GB), which need to be inserted into InnoDB table every min.
I am using LOAD command to accomplish this but in some occasion , the LOAD command takes longer than 1 min. When this happened, the following LOAD file get bigger and bigger and eventually, I get DB gone away error and the program abort.

Any suggestions.
Kye Lee

]]> By: Baron Schwartz http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/#comment-324166 Baron Schwartz Mon, 07 Jul 2008 12:14:10 +0000 http://www.mysqlperformanceblog.com/?p=424#comment-324166 I'm sure you are right Timo. I didn't think it was a signal but I didn't think much about it anyway! I’m sure you are right Timo. I didn’t think it was a signal but I didn’t think much about it anyway!

]]> By: Timo Lindfors http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/#comment-323993 Timo Lindfors Mon, 07 Jul 2008 07:35:26 +0000 http://www.mysqlperformanceblog.com/?p=424#comment-323993 Interesting article. However, isn't "control-d signals EOF" only applicable to terminal devices? If it worked for binary files how could you ever write \x04 to a file? Interesting article. However, isn’t “control-d signals EOF” only applicable to terminal devices? If it worked for binary files how could you ever write \x04 to a file?

]]> By: Baron Schwartz http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/#comment-322639 Baron Schwartz Fri, 04 Jul 2008 15:35:02 +0000 http://www.mysqlperformanceblog.com/?p=424#comment-322639 Pedro, It's a client's machine so I'm not quite sure all the details; but it's an 8-core Intel Xeon L5535 @ 2GHz, 32GB RAM, RAID 10 on 15k SAS drives (I think). Pedro,

It’s a client’s machine so I’m not quite sure all the details; but it’s an 8-core Intel Xeon L5535 @ 2GHz, 32GB RAM, RAID 10 on 15k SAS drives (I think).

]]> By: peter http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/#comment-322358 peter Thu, 03 Jul 2008 23:14:42 +0000 http://www.mysqlperformanceblog.com/?p=424#comment-322358 Baron, Indeed after load is completed you should have significant portion of buffer pool dirty which should take some time to be flushed to the disk. What I would also like to highlight is the slowdown in the log formula happens as data well fits in memory, otherwise you would see number of inserts/sec to drop off through the cliff Baron,

Indeed after load is completed you should have significant portion of buffer pool dirty which should take some time to be flushed to the disk.

What I would also like to highlight is the slowdown in the log formula happens as data well fits in memory, otherwise you would see number of inserts/sec to drop off through the cliff

]]>