Comments on: Living with backups

By: Vitaly

Vitaly — Thu, 02 Jul 2009 12:33:26 +0000

Thank you!
In fact, I’m migrating my backup scripts to another host now.

By: dermoth

dermoth — Thu, 02 Jul 2009 11:43:18 +0000

Vitaly,

That may be a bug…. In 2.6.22 I had one where running I/O-intensive applications in the IDLE class would slow down everything (both MySQL and the IDLE application were running much slower than when they were both in the default class).

Check the crond scripts or use ionice to verify you’re not using classes already (i.e. the system may already use the IDLE class for I/O-intensive jobs)… Other than the my only suggestion would be to upgrade.

By: Vitaly

Vitaly — Thu, 02 Jul 2009 07:46:54 +0000

Thank you, I read your post, probably I just didn’t understand it
First, I’m still with RHEL4 (2.6.9-42.0.3.ELsmp) – it means, I cannot switch between schedulers.
Second – my question is mostly why slowing down is *so* drastic, in many cases it seems like mysql is stucked at all.

Thank you!

By: dermoth

dermoth — Wed, 01 Jul 2009 16:29:24 +0000

Vitaly, you should read my post above… There’s pretty much the answer to your question there.

Nice value affects only processor scheduler and has no effect on IO-intensive tasks. IO classes (ionice) works only with CFQ, and as I point out in that post, even when using IO classes to give priority to MySQL, CFQ makes MySQL run slower than Deadline on my hardware.

I haven’t tested with the Anticipatory scheduler, however it should be even worse by design; Anticipatory should make Mysql wait even more on IO expecting sequential reads; while it will likely make random requests.

Once you find the fastest scheduler for your load/hardware, you may want to tune the scheduler parameters to see if you can get any better results. These are in /sys/block//queue/iosched/.

By: Vitaly

Vitaly — Tue, 30 Jun 2009 09:11:26 +0000

Thanks for the interesting article!
I use my DB server as backup/archiving server for some data (no MySQL related).
I understand that MySQL will respond slowly during when some IO-demanding processes are running (gzip, etc…).
But many times my MySQL DB server just isn’t responding – at least, the same query that usually takes about 2 secs, isn’t responding after few *minutes*!
I played with “ionice for poors” perl script and at the end I’ll probably move all non-DB thins to another box, but I’m just curious – is this some bug in scheduler? in MySQL? or this is expected behaviour?

By: dermoth

dermoth — Tue, 18 Nov 2008 21:02:53 +0000

I played recently with backups while performing server upgrades… My new backup slave performs the whole backup (~60GB uncompressed dump, though using on-the-fly compression with Gzip the file end up slightly over 6GB) in single-transaction while keeping up in replication. Interestingly though, what’s hurting more the server (and sometime tripping off the replication alarms) is the rsync job during the checksuming phase. Quick testing showed that a single process saturating the server with I/O reads cause replicating to slightly lag behind.

I can most likely avoid the checksums in rsync (haven’t tested yet since it’s not really annoying – I think –whole-file does the trick) though I tested various schedulers/setting do see how it affects the results.

Originally running Linux-2.6.22, I had to upgrade to 2.6.27 due to a bug in CFQ: performing read I/O in the Idle was making *both* dd (Idle class) and MySQL (default class) to slow down to ridiculously low speed!

- Using the deadline scheduler (which was the default) gives the best performance in all tested kernels – side read operation are fast and MySQL starts lagging slightly.

- Using CFQ with default class, side read operation is about twice as fast while MySQL replication is nearly halted.

- Using CFQ with side I/O in the Idle class (on 2.6.27), side read operations are still a bit faster than using deadline, and MySQL is lagging behind significantly more than using deadline.

The IO subsystem is a hardware RAID 0+1 (3 stripped RAID1), 6x 10k SCSI-320 disks, 128M controller cache without write caching.

I think that might be of interest to others.

By: Dave Edwards

Dave Edwards — Fri, 14 Nov 2008 19:28:11 +0000

"Replication, backups. We all have to do it. Maciej Dobrzanski of the MySQL Performance Blog expounds on what it means to be living with backups."

By: peter

peter — Fri, 07 Nov 2008 17:26:45 +0000

Using bandwidth limit in rsync or compression speed (with low priority) as a way to offload the system is a good strategy. Though it does not work in all cases. For example if you use rsync to really sync the backup with old backup version quite commonly you will have a lot of disk IO with little bandwidth used to only transfer the data which is actually changed. The compression can lead you into the trouble when you have a lot of small files plus it is not very flexible in picking the pace you want backups to be happening at.

By: peter

peter — Fri, 07 Nov 2008 17:23:34 +0000

Matt,

Using slave for backups is good though you always should consider couple of things

- good replication (and lag monitoring)
- checks to ensure master and slaves are fully in sync.

With maatkit now able to do non-intrusive checks we’ve been checking many replication pairs and the ones which are up and running for a few months quite frequently would have some mild data inconsistencies.

By: Matt

Matt — Fri, 07 Nov 2008 13:29:25 +0000

Maciej,

I used to use gzip because as you said, bzip2 is a lot slower. However with quad core CPUs the parallel bzip2 actually turns out to be faster, even with the -9 flag. Running pbzip2 under ‘nice’ makes sure the DB’s not too starved of CPU time as well although as I’m sure you’re aware, a write-only DB is mostly I/O bound. Also if you’re backing up to a slow storage device (for instance a USB disk or a slow fileserver), pbzip2 can return a substantial time saving as you’re chucking around half the amount of data.