Comments on: How to load large files safely into InnoDB with LOAD DATA INFILE http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/ Percona's MySQL & InnoDB performance and scalability blog Sat, 11 Feb 2012 16:45:54 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: Will http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/comment-page-1/#comment-848635 Will Wed, 30 Nov 2011 01:33:34 +0000 http://www.mysqlperformanceblog.com/?p=424#comment-848635 Old post, but very helpful. We were doing an ignore into load which caused a lot of issues on our production transactions. By splitting up the import into chunks, it eliminated the impact on our production load. Old post, but very helpful. We were doing an ignore into load which caused a lot of issues on our production transactions. By splitting up the import into chunks, it eliminated the impact on our production load.

]]> By: Baron Schwartz http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/comment-page-1/#comment-798270 Baron Schwartz Tue, 15 Feb 2011 18:01:12 +0000 http://www.mysqlperformanceblog.com/?p=424#comment-798270 That's better to ask on the forum, so here's a pointer to the forum :) http://forum.percona.com That’s better to ask on the forum, so here’s a pointer to the forum :) http://forum.percona.com

]]> By: John http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/comment-page-1/#comment-798261 John Tue, 15 Feb 2011 16:59:18 +0000 http://www.mysqlperformanceblog.com/?p=424#comment-798261 Hi, I have run into a little problem with this i have create a bash script to allow me to pass in table name a file to load in data with, this works fine, but if i use the replace option on the load data infile, i get errors of duplicates ERROR 1062 (23000) at line 1: Duplicate entry '85694e353d34b4ab284970f22e3bcd66' for key 'idx_code' any pointers would be really helpful John Hi,

I have run into a little problem with this

i have create a bash script to allow me to pass in table name a file to load in data with, this works fine, but if i use the replace option on the load data infile, i get errors of duplicates

ERROR 1062 (23000) at line 1: Duplicate entry ’85694e353d34b4ab284970f22e3bcd66′ for key ‘idx_code’

any pointers would be really helpful

John

]]> By: Baron Schwartz http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/comment-page-1/#comment-660233 Baron Schwartz Wed, 30 Sep 2009 13:35:05 +0000 http://www.mysqlperformanceblog.com/?p=424#comment-660233 Nishant, I would suspect that you're seeing the tablespace grow because of the insert buffer. This can be controlled in Percona-patched versions of InnoDB, and in XtraDB. Nishant, I would suspect that you’re seeing the tablespace grow because of the insert buffer. This can be controlled in Percona-patched versions of InnoDB, and in XtraDB.

]]> By: Nishant Deshpande http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/comment-page-1/#comment-660167 Nishant Deshpande Wed, 30 Sep 2009 10:45:24 +0000 http://www.mysqlperformanceblog.com/?p=424#comment-660167 Baron, Thanks for the blog as always. I was wondering if this suggests a solution to my problem, namely the shared tablespace (ibdata) file growing even when i have file_per_table and indeed all my tables are created correctly as separate files. when i occasionally do 'insert into new_bigtable select * from bigtable'... i notice that the ibdata file grows huge (unfortunately i haven't run controlled experiements given i only notice this for really large tables 100GB+). i think this also happens when i do a 'load data infile' again we're talking 100GB+ files. Can I make sure I understand your two points above, namely: >> lots of undo log entries collect in the tablespace... from here (http://dev.mysql.com/doc/refman/5.1/en/multiple-tablespaces.html) i see that the undo log entries are kept in the shared tablespace (i'm not sure if you meant that in (1) it wasn't clear to me) so basically if i conduct a transaction on 100GB of data, the ibdata file will necessarily grow to be approximately this size just because i'm doing this as a transaction. once the transaction commits, the undo logs will be 'discarded' but the ibdata file will remain at 100GB. and now i have no way of shrinking this back (unless i do a mysqldump and load which for a large db is prohibitively expensive). as i understand it i can't just copy my .ibd / .frm files and then put them on a new mysql instance. is there any way of avoiding the ibdata file from growing to be as large as the largest transaction effectively? for me the largest transactions would be a data load which would be huge and that means ibdata would be swallowing 20% or more of my disk. Nishant Baron,

Thanks for the blog as always. I was wondering if this suggests a solution to my problem, namely the shared tablespace (ibdata) file growing even when i have file_per_table and indeed all my tables are created correctly as separate files.

when i occasionally do ‘insert into new_bigtable select * from bigtable’… i notice that the ibdata file grows huge (unfortunately i haven’t run controlled experiements given i only notice this for really large tables 100GB+). i think this also happens when i do a ‘load data infile’ again we’re talking 100GB+ files.

Can I make sure I understand your two points above, namely:

>> lots of undo log entries collect in the tablespace…
from here (http://dev.mysql.com/doc/refman/5.1/en/multiple-tablespaces.html) i see that the undo log entries are kept in the shared tablespace (i’m not sure if you meant that in (1) it wasn’t clear to me)

so basically if i conduct a transaction on 100GB of data, the ibdata file will necessarily grow to be approximately this size just because i’m doing this as a transaction. once the transaction commits, the undo logs will be ‘discarded’ but the ibdata file will remain at 100GB. and now i have no way of shrinking this back (unless i do a mysqldump and load which for a large db is prohibitively expensive). as i understand it i can’t just copy my .ibd / .frm files and then put them on a new mysql instance.

is there any way of avoiding the ibdata file from growing to be as large as the largest transaction effectively? for me the largest transactions would be a data load which would be huge and that means ibdata would be swallowing 20% or more of my disk.

Nishant

]]> By: Baron Schwartz http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/comment-page-1/#comment-655040 Baron Schwartz Mon, 21 Sep 2009 01:37:49 +0000 http://www.mysqlperformanceblog.com/?p=424#comment-655040 Hi Gadi, thanks for your comment. I've updated the incorrect code listing and added a bold comment below it. Hi Gadi, thanks for your comment. I’ve updated the incorrect code listing and added a bold comment below it.

]]> By: Gadi Naveh http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/comment-page-1/#comment-654873 Gadi Naveh Sun, 20 Sep 2009 07:26:09 +0000 http://www.mysqlperformanceblog.com/?p=424#comment-654873 Some comments - while the fifo as facility works, it is not obvious from the page that in the loop, the load command must reference the fifo file and NOT the original. it actually says - mysql -e "..... same as above.... " which is misleading. I suggest putting together a step-by-step directions for this page, including a bold comment about which file to use in the load. chrz Some comments – while the fifo as facility works, it is not obvious from the page that in the loop, the load command must reference the fifo file and NOT the original. it actually says – mysql -e “….. same as above…. ” which is misleading.

I suggest putting together a step-by-step directions for this page, including a bold comment about which file to use in the load.

chrz

]]> By: Baron Schwartz http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/comment-page-1/#comment-359384 Baron Schwartz Fri, 03 Oct 2008 13:28:06 +0000 http://www.mysqlperformanceblog.com/?p=424#comment-359384 Only MyISAM supports full-text indexing in MySQL. If you have a lot of content to index (bigger than your available memory) and you need high performance, you probably need an external solution such as Sphinx or Lucene. Sphinx has a storage engine for easy integration with MySQL. Only MyISAM supports full-text indexing in MySQL. If you have a lot of content to index (bigger than your available memory) and you need high performance, you probably need an external solution such as Sphinx or Lucene. Sphinx has a storage engine for easy integration with MySQL.

]]> By: LeRoy Grubbs http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/comment-page-1/#comment-359223 LeRoy Grubbs Thu, 02 Oct 2008 21:24:27 +0000 http://www.mysqlperformanceblog.com/?p=424#comment-359223 What's the best way to load lots of large and small files for full text indexing? Which database engine is best suited for FTI? of large files? What’s the best way to load lots of large and small files for full text indexing? Which database engine is best suited for FTI? of large files?

]]> By: Baron Schwartz http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/comment-page-1/#comment-336167 Baron Schwartz Mon, 28 Jul 2008 13:13:57 +0000 http://www.mysqlperformanceblog.com/?p=424#comment-336167 Hi Kye, Please use the Contact Us form on our website http://www.percona.com, as this goes into our ticketing system. Hi Kye,

Please use the Contact Us form on our website http://www.percona.com, as this goes into our ticketing system.

]]>