Comments on: How to load large files safely into InnoDB with LOAD DATA INFILE http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/ Everything about MySQL Performance Sat, 21 Nov 2009 05:23:57 -0800 http://wordpress.org/?v=2.8.4 hourly 1 By: Baron Schwartz http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/comment-page-1/#comment-660233 Baron Schwartz Wed, 30 Sep 2009 13:35:05 +0000 http://www.mysqlperformanceblog.com/?p=424#comment-660233 Nishant, I would suspect that you're seeing the tablespace grow because of the insert buffer. This can be controlled in Percona-patched versions of InnoDB, and in XtraDB. Nishant, I would suspect that you’re seeing the tablespace grow because of the insert buffer. This can be controlled in Percona-patched versions of InnoDB, and in XtraDB.

]]> By: Nishant Deshpande http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/comment-page-1/#comment-660167 Nishant Deshpande Wed, 30 Sep 2009 10:45:24 +0000 http://www.mysqlperformanceblog.com/?p=424#comment-660167 Baron, Thanks for the blog as always. I was wondering if this suggests a solution to my problem, namely the shared tablespace (ibdata) file growing even when i have file_per_table and indeed all my tables are created correctly as separate files. when i occasionally do 'insert into new_bigtable select * from bigtable'... i notice that the ibdata file grows huge (unfortunately i haven't run controlled experiements given i only notice this for really large tables 100GB+). i think this also happens when i do a 'load data infile' again we're talking 100GB+ files. Can I make sure I understand your two points above, namely: >> lots of undo log entries collect in the tablespace... from here (http://dev.mysql.com/doc/refman/5.1/en/multiple-tablespaces.html) i see that the undo log entries are kept in the shared tablespace (i'm not sure if you meant that in (1) it wasn't clear to me) so basically if i conduct a transaction on 100GB of data, the ibdata file will necessarily grow to be approximately this size just because i'm doing this as a transaction. once the transaction commits, the undo logs will be 'discarded' but the ibdata file will remain at 100GB. and now i have no way of shrinking this back (unless i do a mysqldump and load which for a large db is prohibitively expensive). as i understand it i can't just copy my .ibd / .frm files and then put them on a new mysql instance. is there any way of avoiding the ibdata file from growing to be as large as the largest transaction effectively? for me the largest transactions would be a data load which would be huge and that means ibdata would be swallowing 20% or more of my disk. Nishant Baron,

Thanks for the blog as always. I was wondering if this suggests a solution to my problem, namely the shared tablespace (ibdata) file growing even when i have file_per_table and indeed all my tables are created correctly as separate files.

when i occasionally do ‘insert into new_bigtable select * from bigtable’… i notice that the ibdata file grows huge (unfortunately i haven’t run controlled experiements given i only notice this for really large tables 100GB+). i think this also happens when i do a ‘load data infile’ again we’re talking 100GB+ files.

Can I make sure I understand your two points above, namely:

>> lots of undo log entries collect in the tablespace…
from here (http://dev.mysql.com/doc/refman/5.1/en/multiple-tablespaces.html) i see that the undo log entries are kept in the shared tablespace (i’m not sure if you meant that in (1) it wasn’t clear to me)

so basically if i conduct a transaction on 100GB of data, the ibdata file will necessarily grow to be approximately this size just because i’m doing this as a transaction. once the transaction commits, the undo logs will be ‘discarded’ but the ibdata file will remain at 100GB. and now i have no way of shrinking this back (unless i do a mysqldump and load which for a large db is prohibitively expensive). as i understand it i can’t just copy my .ibd / .frm files and then put them on a new mysql instance.

is there any way of avoiding the ibdata file from growing to be as large as the largest transaction effectively? for me the largest transactions would be a data load which would be huge and that means ibdata would be swallowing 20% or more of my disk.

Nishant

]]> By: Baron Schwartz http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/comment-page-1/#comment-655040 Baron Schwartz Mon, 21 Sep 2009 01:37:49 +0000 http://www.mysqlperformanceblog.com/?p=424#comment-655040 Hi Gadi, thanks for your comment. I've updated the incorrect code listing and added a bold comment below it. Hi Gadi, thanks for your comment. I’ve updated the incorrect code listing and added a bold comment below it.

]]> By: Gadi Naveh http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/comment-page-1/#comment-654873 Gadi Naveh Sun, 20 Sep 2009 07:26:09 +0000 http://www.mysqlperformanceblog.com/?p=424#comment-654873 Some comments - while the fifo as facility works, it is not obvious from the page that in the loop, the load command must reference the fifo file and NOT the original. it actually says - mysql -e "..... same as above.... " which is misleading. I suggest putting together a step-by-step directions for this page, including a bold comment about which file to use in the load. chrz Some comments – while the fifo as facility works, it is not obvious from the page that in the loop, the load command must reference the fifo file and NOT the original. it actually says – mysql -e “….. same as above…. ” which is misleading.

I suggest putting together a step-by-step directions for this page, including a bold comment about which file to use in the load.

chrz

]]> By: Baron Schwartz http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/comment-page-1/#comment-359384 Baron Schwartz Fri, 03 Oct 2008 13:28:06 +0000 http://www.mysqlperformanceblog.com/?p=424#comment-359384 Only MyISAM supports full-text indexing in MySQL. If you have a lot of content to index (bigger than your available memory) and you need high performance, you probably need an external solution such as Sphinx or Lucene. Sphinx has a storage engine for easy integration with MySQL. Only MyISAM supports full-text indexing in MySQL. If you have a lot of content to index (bigger than your available memory) and you need high performance, you probably need an external solution such as Sphinx or Lucene. Sphinx has a storage engine for easy integration with MySQL.

]]> By: LeRoy Grubbs http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/comment-page-1/#comment-359223 LeRoy Grubbs Thu, 02 Oct 2008 21:24:27 +0000 http://www.mysqlperformanceblog.com/?p=424#comment-359223 What's the best way to load lots of large and small files for full text indexing? Which database engine is best suited for FTI? of large files? What’s the best way to load lots of large and small files for full text indexing? Which database engine is best suited for FTI? of large files?

]]> By: Baron Schwartz http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/comment-page-1/#comment-336167 Baron Schwartz Mon, 28 Jul 2008 13:13:57 +0000 http://www.mysqlperformanceblog.com/?p=424#comment-336167 Hi Kye, Please use the Contact Us form on our website http://www.percona.com, as this goes into our ticketing system. Hi Kye,

Please use the Contact Us form on our website http://www.percona.com, as this goes into our ticketing system.

]]> By: Kye Lee http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/comment-page-1/#comment-336165 Kye Lee Mon, 28 Jul 2008 13:11:59 +0000 http://www.mysqlperformanceblog.com/?p=424#comment-336165 Please send me the private email with contact info. Thanks Kye Lee Please send me the private email with contact info.

Thanks
Kye Lee

]]> By: Baron Schwartz http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/comment-page-1/#comment-334731 Baron Schwartz Fri, 25 Jul 2008 23:29:35 +0000 http://www.mysqlperformanceblog.com/?p=424#comment-334731 I would suggest breaking it into smaller pieces, but it sounds like you have other problems and need a completely different approach -- perhaps the problem is that you even need these bulk loads. Beyond that, I won't say; this is what we do for a living :-) I would suggest breaking it into smaller pieces, but it sounds like you have other problems and need a completely different approach — perhaps the problem is that you even need these bulk loads. Beyond that, I won’t say; this is what we do for a living :-)

]]> By: Kye Lee http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/comment-page-1/#comment-334538 Kye Lee Fri, 25 Jul 2008 15:48:23 +0000 http://www.mysqlperformanceblog.com/?p=424#comment-334538 I found your article and thought it is very interesting – Thanks. As result of your test, do you have recommendation or method of efficiently loading very large file? BTW – I am NOT heavy DB programmer and don’t know much about DB. If you don’t mine I would like to seek your advice and help. I have about 120,000 rows – rec size 130 bytes with about 13 fields (Avg 15 GB), which need to be inserted into InnoDB table every min. I am using LOAD command to accomplish this but in some occasion , the LOAD command takes longer than 1 min. When this happened, the following LOAD file get bigger and bigger and eventually, I get DB gone away error and the program abort. Any suggestions. Kye Lee I found your article and thought it is very interesting – Thanks.
As result of your test, do you have recommendation or method of efficiently loading very large file?

BTW – I am NOT heavy DB programmer and don’t know much about DB. If you don’t mine I would like to seek your advice and help.

I have about 120,000 rows – rec size 130 bytes with about 13 fields (Avg 15 GB), which need to be inserted into InnoDB table every min.
I am using LOAD command to accomplish this but in some occasion , the LOAD command takes longer than 1 min. When this happened, the following LOAD file get bigger and bigger and eventually, I get DB gone away error and the program abort.

Any suggestions.
Kye Lee

]]>