Comments on: Wishes for mysqldump http://www.mysqlperformanceblog.com/2007/05/22/wishes-for-mysqldump/ Everything about MySQL Performance Sun, 07 Sep 2008 08:30:14 +0000 http://wordpress.org/?v=2.5.1 By: Thomas http://www.mysqlperformanceblog.com/2007/05/22/wishes-for-mysqldump/#comment-306613 Thomas Thu, 29 May 2008 19:41:45 +0000 http://www.mysqlperformanceblog.com/2007/05/22/wishes-for-mysqldump/#comment-306613 3. I wrote a simple perl script that does just that... I only tested with 16 threads so far (on a 16-proc server) and IO seek seemed to me a big problem (the restore was probably between 50 and 200 times slower! - I just aborted it). It's not a production RAID (only for local C drive, "value" kind of hardware...) and I may do more tests on a real RAID. My script is a bit crappy; it buffers tables in memory then send them in one block to child processes, and block all threads on db changes. I know how I can make it better by using queues and non-blocking I/O but I'm unlikely to to this anytime soon due to time constraints. It's fairly simple to do with Event::Lib though. Also I'm not sure if the specs for dumps are published... I rely solely on comments in the dump and on the fact that global parameters are only present at the start of it. Also I based my work on a specific database dump; If there's a spec somewhere it could help. 3. I wrote a simple perl script that does just that… I only tested with 16 threads so far (on a 16-proc server) and IO seek seemed to me a big problem (the restore was probably between 50 and 200 times slower! - I just aborted it). It’s not a production RAID (only for local C drive, “value” kind of hardware…) and I may do more tests on a real RAID.

My script is a bit crappy; it buffers tables in memory then send them in one block to child processes, and block all threads on db changes. I know how I can make it better by using queues and non-blocking I/O but I’m unlikely to to this anytime soon due to time constraints. It’s fairly simple to do with Event::Lib though.

Also I’m not sure if the specs for dumps are published… I rely solely on comments in the dump and on the fact that global parameters are only present at the start of it. Also I based my work on a specific database dump; If there’s a spec somewhere it could help.

]]> By: Introducing MySQL Parallel Dump at Xaprb http://www.mysqlperformanceblog.com/2007/05/22/wishes-for-mysqldump/#comment-173985 Introducing MySQL Parallel Dump at Xaprb Mon, 01 Oct 2007 02:18:18 +0000 http://www.mysqlperformanceblog.com/2007/05/22/wishes-for-mysqldump/#comment-173985 [...] while ago Peter Zaitsev wrote about his wishes for mysqldump. These included multi-threaded dumps and “safe” dumps that would wait for a server to [...] [...] while ago Peter Zaitsev wrote about his wishes for mysqldump. These included multi-threaded dumps and “safe” dumps that would wait for a server to [...]

]]> By: Mark Callaghan http://www.mysqlperformanceblog.com/2007/05/22/wishes-for-mysqldump/#comment-130656 Mark Callaghan Thu, 24 May 2007 17:15:59 +0000 http://www.mysqlperformanceblog.com/2007/05/22/wishes-for-mysqldump/#comment-130656 Chip released compact_innodb.py many months ago. Despite the _compact_innodb_, it should work with more storage engines. It does parallel load and restore by forking mysqldump and mysql processes. The project page for the code is http://code.google.com/p/google-mysql-tools. Chip released compact_innodb.py many months ago. Despite the _compact_innodb_, it should work with more storage engines. It does parallel load and restore by forking mysqldump and mysql processes. The project page for the code is http://code.google.com/p/google-mysql-tools.

]]> By: fr3nd http://www.mysqlperformanceblog.com/2007/05/22/wishes-for-mysqldump/#comment-130645 fr3nd Thu, 24 May 2007 16:05:47 +0000 http://www.mysqlperformanceblog.com/2007/05/22/wishes-for-mysqldump/#comment-130645 Hey, I've released a new version of mysqlpdump whith some important improvements: * Store dumps to files directly instead to stdout * Can compress files * Dump each table in its own file * Can pass parameters directly to mysqldump http://www.fr3nd.net/projects/mysqlpdump Next thing I want to implement is to store all the replication data to a separate .sql file. Any suggestion? Hey,

I’ve released a new version of mysqlpdump whith some important improvements:

* Store dumps to files directly instead to stdout
* Can compress files
* Dump each table in its own file
* Can pass parameters directly to mysqldump

http://www.fr3nd.net/projects/mysqlpdump

Next thing I want to implement is to store all the replication data to a separate .sql file.

Any suggestion?

]]> By: flupps http://www.mysqlperformanceblog.com/2007/05/22/wishes-for-mysqldump/#comment-130570 flupps Thu, 24 May 2007 03:56:25 +0000 http://www.mysqlperformanceblog.com/2007/05/22/wishes-for-mysqldump/#comment-130570 Brian, Larry: There's been a patch around for this since 2005, unfortunately never got pushed into the mysqldump... http://bugs.mysql.com/file.php?id=959 Brian, Larry:

There’s been a patch around for this since 2005, unfortunately never got pushed into the mysqldump…
http://bugs.mysql.com/file.php?id=959

]]> By: peter http://www.mysqlperformanceblog.com/2007/05/22/wishes-for-mysqldump/#comment-130497 peter Wed, 23 May 2007 17:36:30 +0000 http://www.mysqlperformanceblog.com/2007/05/22/wishes-for-mysqldump/#comment-130497 Thank Brian, Really I like tab format a lot. LOAD DATA INFILE should be easier to optimize for storage engines internally if they like to. Right now I think it is underused because you need to process each database separately. Thank Brian,

Really I like tab format a lot. LOAD DATA INFILE should be easier to optimize for storage engines internally if they like to. Right now I think it is underused because you need to process each database separately.

]]> By: Brian Aker http://www.mysqlperformanceblog.com/2007/05/22/wishes-for-mysqldump/#comment-130464 Brian Aker Wed, 23 May 2007 16:51:27 +0000 http://www.mysqlperformanceblog.com/2007/05/22/wishes-for-mysqldump/#comment-130464 Hi! I've been thinking about a --clone option for a bit now... allow you to get the slave options plus take the name of a machine to feed the data to. Bypass the need to pump data via: mysqldump [args] | mysql --host --port Which is something I do all of the time to make new slaves. If I get a spare minute I'll look at your idea on parallel dump with locks. If I do it though, I would be interested in it via a tab dump. Cheers, -Brian Hi!

I’ve been thinking about a –clone option for a bit now… allow you to get the slave options plus take the name of a machine to feed the data to. Bypass the need to pump data via:
mysqldump [args] | mysql –host –port

Which is something I do all of the time to make new slaves. If I get a spare minute I’ll look at your idea on parallel dump with locks. If I do it though, I would be interested in it via a tab dump.

Cheers,
-Brian

]]> By: fr3nd http://www.mysqlperformanceblog.com/2007/05/22/wishes-for-mysqldump/#comment-130442 fr3nd Wed, 23 May 2007 16:21:32 +0000 http://www.mysqlperformanceblog.com/2007/05/22/wishes-for-mysqldump/#comment-130442 I wrote a quick and dirty wrapper in python that launches many threads with mysqldump. Each thread stores the result in memory and waits for the preceding one. That's the reason why it could be so memory consuming with databases with large tables. You can find it here: http://www.fr3nd.net/projects/mysqlpdump It's pretty raw at the moment, but it can be used as skeleton to build something more complex. Any comments will be appreciated :) I wrote a quick and dirty wrapper in python that launches many threads with mysqldump. Each thread stores the result in memory and waits for the preceding one. That’s the reason why it could be so memory consuming with databases with large tables.

You can find it here: http://www.fr3nd.net/projects/mysqlpdump

It’s pretty raw at the moment, but it can be used as skeleton to build something more complex.

Any comments will be appreciated :)

]]> By: peter http://www.mysqlperformanceblog.com/2007/05/22/wishes-for-mysqldump/#comment-130380 peter Wed, 23 May 2007 11:56:39 +0000 http://www.mysqlperformanceblog.com/2007/05/22/wishes-for-mysqldump/#comment-130380 Larry, Absolutely. This is why in MMM backup tools we store both SHOW SLAVE STATUS and SHOW MASTER STATUS info so you can when use it to restore master and do point in time recovery using binary log or restore slave and get it going. Larry,

Absolutely. This is why in MMM backup tools we store both SHOW SLAVE STATUS and SHOW MASTER STATUS info so you can when use it to restore master and do point in time recovery using binary log or restore slave and get it going.

]]> By: Larry Damman http://www.mysqlperformanceblog.com/2007/05/22/wishes-for-mysqldump/#comment-130222 Larry Damman Wed, 23 May 2007 08:56:01 +0000 http://www.mysqlperformanceblog.com/2007/05/22/wishes-for-mysqldump/#comment-130222 While we're requesting features, I have one that I'd love to see implemented. It would be similar to --master-data=1 except it would get the Exec_Master_Log_Pos from the "SHOW SLAVE STATUS" command. This way you could (easily) create a sibling of the dumped db instead of a child. This would come in handy when you take your backups/snapshots off of a slave in order to keep the master running smoothly. While we’re requesting features, I have one that I’d love to see implemented. It would be similar to –master-data=1 except it would get the Exec_Master_Log_Pos from the “SHOW SLAVE STATUS” command. This way you could (easily) create a sibling of the dumped db instead of a child. This would come in handy when you take your backups/snapshots off of a slave in order to keep the master running smoothly.

]]>