Brian, I was told that LOAD DATA works differently now. Quote from Seppo: “LOAD DATA INFILE processing will commit every 10K rows. So large transactions due to LOAD DATA will be split to series of small transactions”. We should fix documentation.
Post: How to load large files safely into InnoDB with LOAD DATA INFILE
July 3, 2008 By 28 Comments
… a customer ask me about loading two huge files into InnoDB with LOAD DATA INFILE. The goal was to load this data on many servers without… very large transaction caused by the single statement. We didn’t want to split the file into pieces for the load for… 1; done Note that the file mentioned in LOAD DATA INFILE is /tmp/my-fifo, not infile.txt! After I was done, I ran…
Post: Predicting how long data load would take
May 24, 2007 By 12 Comments
…needs large innodb_buffer_pool_size and large innodb_log_file_size to perform load effectively Load Options There two main ways to load data in MySQL you can use Multiple value insert (standard mysqldump output) or LOAD DATA INFILE (–tab mysqldump output). Generally LOAD DATA can …
Post: Quick comparison of MyISAM, Infobright, and MonetDB
September 29, 2009 By 30 Comments
… doing was loading the data with SQL statements. I wanted to see how fast MyISAM vs. MonetDB would interpret really large INSERT statements…. I used LOAD DATA INFILE instead (MonetDB’s version of that is COPY INTO). This is the only way to get data into Infobright… different optimizer, etc. This point was hammered home during the LOAD DATA INFILE, when I looked to see what was taking so long…
Post: High-Performance Click Analysis with MySQL
December 22, 2008 By 13 Comments
… reason denormalization is important is that nested-loop joins on large data sets are very expensive. If MySQL supported sort-merge… it write out a CSV file and import that with LOAD DATA INFILE. Keep those big fat log files out of the…, then piping the results back up to the master with LOAD DATA INFILE, which kind of emulates row-based replication in a way…
Post: Resyncing table on MySQL Slave
June 29, 2008 By 10 Comments
… * FROM table INTO OUTFILE ‘/tmp/tbl.txt’; DELETE FROM tbl; LOAD DATA INFILE ‘tmp/tbl.txt’ INTO TABLE tbl; UNLOCK TABLES; This will…_maint; Query OK, 0 rows affected (0.00 sec) mysql> load data infile ‘/tmp/rep.txt’ into table rep_new; Query OK, 0… master and slave. If you need quick resync of large amount of data it is much better to use LVM snapshots or…
Comment: How to load large files safely into InnoDB with LOAD DATA INFILE
September 30, 2009 By 28 Comments
… only notice this for really large tables 100GB+). i think this also happens when i do a ‘load data infile‘ again we’re talking… shrinking this back (unless i do a mysqldump and load which for a large db is prohibitively expensive). as i understand it… growing to be as large as the largest transaction effectively? for me the largest transactions would be a data load which would be…
Post: Using any general purpose computer as a special purpose SIMD computer
May 16, 2011 By 6 Comments
… very large. I inflated the document size significantly to over 3 million “words” by duplicating the entire set multiple times. mysql> load data infile…, not relations. If it were smarter it would ask the data dictionary about partitioning. Array ( [word] => Congress: [md5(word)] => 873c419d2c2139bc8bbc3cbaffcc3473 [md5…
Comment: Quick comparison of MyISAM, Infobright, and MonetDB
September 30, 2009 By 30 Comments
…, I’m surprised by those LOAD DATA INFILE results. I didn’t catch how many rows you ended up loading, but from the size… useful only in the form of INSERT INTO .. SELECT .. and large batches, and even so, going through a loader file or… easy typical case is incremental time-series data where the timestamp essentially automatically partitions the data set. EXPLAIN out of Infobright is…
Post: MySQL Users Conference - Innodb
May 9, 2007 By 6 Comments
… does which can be multiple orders of magnitude faster for large tables. There is fair number of questions about this feature… I do not yet know answer for – for example will LOAD DATA INFILE be optimized same way as for MyISAM tables by separate… should be much faster for IO Bound Index scans and large range scans. We surely should benchmark this aspect separately. Improved…
