Comments on: How would you compress your MySQL Backup

By: Steven

Steven — Tue, 26 Jan 2010 05:52:33 +0000

Would be nice to see rzip / lrzip from tridge. http://rzip.samba.org/

By: Serge Knystautas

Serge Knystautas — Wed, 29 Oct 2008 16:40:44 +0000

I saw the LZO SMP reference, so thought I’d look up the GZIP equivalent…

My servers are all multicores and I’m wondering with my next backup if I can’t accelerate GZIP significantly by spreading is across multiple cores and get closer to that 100MB/s SATA speed.

By: Lasse Reinhold

Lasse Reinhold — Wed, 25 Jun 2008 22:12:51 +0000

I’m creating a high performance file archiver, qpress, which uses asynchronous I/O (reader and writer thread). On Windows it’s also non-buffered to prevent the cache of other server applications from being flushed. It’s using the QuickLZ 1.40 compression library and is intended for speed, not good compression ratio.

I’d be glad for any feedback on performance, bugs or ideas for new feature. Download a prototype at:

http://www.quicklz.com/qpress.zip

By: joel

joel — Tue, 24 Jun 2008 21:37:06 +0000

I actually ran into this problem a few days ago. While taring a lvm snapshot one of my cores was pegged at 100%. After a bit of googleling I found the above mentioned pigz. Using it I cut my backup time to 1/4 of the original time. Backups with pigz are restorable with regular old gzip. There is also a parallel bzip2 out there, pbzip2, but I did not try it as it does not accept stdin.

By: art

art — Mon, 16 Jun 2008 09:45:45 +0000

What about rar?

By: peter

peter — Fri, 13 Jun 2008 16:45:02 +0000

Mamatha,

The compression ration depends on data a lot. For all compressors I tested I specified the result file size. Raw file was 1GB.

By: Mamatha

Mamatha — Fri, 13 Jun 2008 08:49:40 +0000

I wanted to know the compression ratio of LZO. IF I have a file of 12 MB , what is the size of the compressed file?

By: jeffatrackaid

jeffatrackaid — Tue, 10 Jun 2008 17:04:23 +0000

We are usually pulling backups from slaves who’s main purpose is backups, so I’ve not done may benchmarks. But on some standalone boxes where I have to compress a large amount of data, I sometimes nice my process to a friendly level. Not sure if this really makes much of an impact, just a habit I developed. There is another interesting tool called 7zip (http://www.7-zip.org/). Windows and Linux friendly. I’ve not benchmarked it for speed but the native format does very well on HTML files — pushed 3GBs of HTML into just 30MB.

By: Kyo

Kyo — Sun, 08 Jun 2008 18:28:38 +0000

I consider bzip2 obsolete for all practical purposes, at least as far as my machines are involved. LZMA is so much faster (especially at decompressing) while often compressing even much better than bzip2 that the only use of bzip2 that I still have is for exchanging data with the outside world. If you are not too picky about the format (I am not, at least for archival purposes) and very demanding as far as speed is concerned, p7zip wonderfully exploits multiple cores for both compression and decompression – try it on a quad core machine. Other than that, GNU tar now also integrates LZMA, as far as I know. And SUSE, for example. decided to go with LZMA in RPM payload in the upcoming release of their distro. The packages are often much smaller and install somewhere from two to three times faster.

By: peter

peter — Sun, 08 Jun 2008 17:50:01 +0000

Pat,

If you’re using MyISAM you need to do:

FLUSH TABLES WITH READ LOCK;

UNLOCK TABLES

This means there will be short (or long) read only moment but there will be no MyISAM corruption.

BTW – tools like mylvmbackup already can do this for you and implement various tricks to make the read only time shorter.