Comments on: Using netcat to copy MySQL Database http://www.mysqlperformanceblog.com/2009/05/31/using-netcat-to-copy-mysql-database/ Everything about MySQL Performance Sat, 21 Nov 2009 05:23:57 -0800 http://wordpress.org/?v=2.8.4 hourly 1 By: James Byers http://www.mysqlperformanceblog.com/2009/05/31/using-netcat-to-copy-mysql-database/comment-page-1/#comment-660021 James Byers Wed, 30 Sep 2009 05:56:28 +0000 http://www.mysqlperformanceblog.com/?p=697#comment-660021 If using Ken's rsync example and you're root on both sides, you'll likely need: uid = root gid = root above [db] in the rsync config file to avoid permission problems. If using Ken’s rsync example and you’re root on both sides, you’ll likely need:

uid = root
gid = root

above

[db]

in the rsync config file to avoid permission problems.

]]> By: Morgan Christiansson http://www.mysqlperformanceblog.com/2009/05/31/using-netcat-to-copy-mysql-database/comment-page-1/#comment-586308 Morgan Christiansson Mon, 15 Jun 2009 20:17:40 +0000 http://www.mysqlperformanceblog.com/?p=697#comment-586308 It is possible to use a less expensive cipher with ssh which will speed things up significantly. For example: rsync -e 'ssh -c blowfish' ... scp -c blowfish ... It is possible to use a less expensive cipher with ssh which will speed things up significantly.

For example:
rsync -e ’ssh -c blowfish’ …
scp -c blowfish …

]]> By: Kevin Burton http://www.mysqlperformanceblog.com/2009/05/31/using-netcat-to-copy-mysql-database/comment-page-1/#comment-578387 Kevin Burton Mon, 08 Jun 2009 04:38:01 +0000 http://www.mysqlperformanceblog.com/?p=697#comment-578387 I agree with #1 .... do NOT do this on production data without checksums. You WILL get data corruption. I agree with #1 …. do NOT do this on production data without checksums.

You WILL get data corruption.

]]> By: Ken http://www.mysqlperformanceblog.com/2009/05/31/using-netcat-to-copy-mysql-database/comment-page-1/#comment-575777 Ken Fri, 05 Jun 2009 19:58:39 +0000 http://www.mysqlperformanceblog.com/?p=697#comment-575777 RT: agreed. Przemek: 1. You'll find more and more that these days people using MySQL run HA systems (as HA as MySQL gets anyway) for international users and systems, not just a little PHP site in Dubuque. The "night-time trough" is no longer a luxury for a lot of the client base of MySQL. 2. LVM snapshot overhead alone can be 20-50%; startups with limited capex don't want to pay 20-50% more for their storage hardware to support a method of database backup used solely because a working solution isn't available from the DB or engine vendor (explain /that/ to a VC). Established companies don't want to budget in 20-50% more for their expensive SAN setups. We'd all love a few thousand spare TPS or a few 100MB/s of spare throughput, but that money should be going to what makes money, not what loses money. This also doesn't cover the overhead of saturating the disk subsystem with the *actual copy* with tar/nc or rsync. Been there. Done both. Bought the T-shirt. 3. Absolutely agree; the many other benefits of a slave makes the cost worthwhile. LVM is a very useful tool, but the fact that it is often represented as a panacea by the MySQL community (not singling you out) does a disservice to those relying on that information (who then have it blow up in their face). IMHO; apologies for the rant. -- Ken. RT: agreed.

Przemek:

1. You’ll find more and more that these days people using MySQL run HA systems (as HA as MySQL gets anyway) for international users and systems, not just a little PHP site in Dubuque. The “night-time trough” is no longer a luxury for a lot of the client base of MySQL.

2. LVM snapshot overhead alone can be 20-50%; startups with limited capex don’t want to pay 20-50% more for their storage hardware to support a method of database backup used solely because a working solution isn’t available from the DB or engine vendor (explain /that/ to a VC). Established companies don’t want to budget in 20-50% more for their expensive SAN setups. We’d all love a few thousand spare TPS or a few 100MB/s of spare throughput, but that money should be going to what makes money, not what loses money. This also doesn’t cover the overhead of saturating the disk subsystem with the *actual copy* with tar/nc or rsync. Been there. Done both. Bought the T-shirt.

3. Absolutely agree; the many other benefits of a slave makes the cost worthwhile.

LVM is a very useful tool, but the fact that it is often represented as a panacea by the MySQL community (not singling you out) does a disservice to those relying on that information (who then have it blow up in their face). IMHO; apologies for the rant.
–
Ken.

]]> By: Przemek http://www.mysqlperformanceblog.com/2009/05/31/using-netcat-to-copy-mysql-database/comment-page-1/#comment-575731 Przemek Fri, 05 Jun 2009 19:01:29 +0000 http://www.mysqlperformanceblog.com/?p=697#comment-575731 1. You do it at off peak time. 2. IMHO if your system dies only because of LVM snapshot overhead - you're far too close to your machine's disk I/O limits. 3. IMO always the best solution is to have a spare slave used only for backups, some periodic heavy queries, etc. 1. You do it at off peak time.
2. IMHO if your system dies only because of LVM snapshot overhead – you’re far too close to your machine’s disk I/O limits.
3. IMO always the best solution is to have a spare slave used only for backups, some periodic heavy queries, etc.

]]> By: Richard Thomas http://www.mysqlperformanceblog.com/2009/05/31/using-netcat-to-copy-mysql-database/comment-page-1/#comment-575728 Richard Thomas Fri, 05 Jun 2009 18:49:39 +0000 http://www.mysqlperformanceblog.com/?p=697#comment-575728 As its even been reported on this blog, the LVM snapshot penalty can be extreme, on a overly busy server (and yes I understand thats a problem of its own) starting up an LVM snapshot can cause a cascading failure of the system as it struggles to keep up. As its even been reported on this blog, the LVM snapshot penalty can be extreme, on a overly busy server (and yes I understand thats a problem of its own) starting up an LVM snapshot can cause a cascading failure of the system as it struggles to keep up.

]]> By: Przemek http://www.mysqlperformanceblog.com/2009/05/31/using-netcat-to-copy-mysql-database/comment-page-1/#comment-575719 Przemek Fri, 05 Jun 2009 18:42:40 +0000 http://www.mysqlperformanceblog.com/?p=697#comment-575719 Instead of shutdowning master just use LVM snapshot method to create a slave or backup. This way you only need a few secs of read lock. If you don't use LVM partitions I suggest to start using them (optimal separate ones for data and binlogs). Instead of shutdowning master just use LVM snapshot method to create a slave or backup. This way you only need a few secs of read lock.
If you don’t use LVM partitions I suggest to start using them (optimal separate ones for data and binlogs).

]]> By: Richard Thomas http://www.mysqlperformanceblog.com/2009/05/31/using-netcat-to-copy-mysql-database/comment-page-1/#comment-575704 Richard Thomas Fri, 05 Jun 2009 18:18:44 +0000 http://www.mysqlperformanceblog.com/?p=697#comment-575704 Rsync is great for live replication copies also 1. rsync with database running, depending on last rsync can take a while 2. Flush tables/cache, rsync again, won't take nearly as long 3. shutdown master 4. rysnc one final time, on a 10gig db only takes a couple seconds 4. restart master, downtime only a few seconds and you have a fresh copy to use for starting a slave back up Of course you need to delete the proper log files depending on your setup and how you store your log files. Rsync is great for live replication copies also

1. rsync with database running, depending on last rsync can take a while
2. Flush tables/cache, rsync again, won’t take nearly as long
3. shutdown master
4. rysnc one final time, on a 10gig db only takes a couple seconds
4. restart master, downtime only a few seconds and you have a fresh copy to use for starting a slave back up

Of course you need to delete the proper log files depending on your setup and how you store your log files.

]]> By: MattW http://www.mysqlperformanceblog.com/2009/05/31/using-netcat-to-copy-mysql-database/comment-page-1/#comment-573417 MattW Wed, 03 Jun 2009 16:40:04 +0000 http://www.mysqlperformanceblog.com/?p=697#comment-573417 in the utility copy vein, if you have a source host, a target host on rfc1918 address space, and an intermediate bastion host you have to ssh to in order to REACH the target host (since it doesn't have an internet-legal ip) and you don't have a vpn/etc, you can pipe tar through a chained ssh via: tar cvf - mysql_dir | ssh -A user@bastionhost.example.com "ssh -A user@rfc1918.example.com 'cd /path/to/target/;tar xvf -'" in the utility copy vein, if you have a source host, a target host on rfc1918 address space, and an intermediate bastion host you have to ssh to in order to REACH the target host (since it doesn’t have an internet-legal ip) and you don’t have a vpn/etc, you can pipe tar through a chained ssh via:

tar cvf – mysql_dir | ssh -A user@bastionhost.example.com “ssh -A user@rfc1918.example.com ‘cd /path/to/target/;tar xvf -’”

]]> By: Joel K. http://www.mysqlperformanceblog.com/2009/05/31/using-netcat-to-copy-mysql-database/comment-page-1/#comment-572417 Joel K. Tue, 02 Jun 2009 16:04:04 +0000 http://www.mysqlperformanceblog.com/?p=697#comment-572417 My Ubuntu 8.04 system didn't know about bar. I've always used 'pv' for essentially the same function. pv - monitor the progress of data through a pipe Which appears quite similar. My Ubuntu 8.04 system didn’t know about bar.

I’ve always used ‘pv’ for essentially the same function.

pv – monitor the progress of data through a pipe

Which appears quite similar.

]]>