Comments on: Living with backups

By: Kattare

Kattare — Fri, 19 Feb 2010 21:53:28 +0000

Baron, I went ahead and did some tests. The files were coming out with different md5sums. I expanded my tests to different server model pairs and discovered that some servers worked, others didn’t. They’re all running the same OS, so I’m not sure what the common thread is yet. Whether it’s a cpu issue, mobo issue, memory issue, or “2.6.XX kernel with XX options compiled in on XX hardware” kind of issue. We run a lot of commodity hardware here so it’s not uncommon to find oddities every here and there, but it’s been a while now since I’ve run into one like this.

By: Baron Schwartz

Baron Schwartz — Fri, 22 Jan 2010 03:03:15 +0000

Kattare, I see. Have you tried to md5sum files to check if they are the same after rsync?

By: Kattare

Kattare — Thu, 21 Jan 2010 18:34:44 +0000

Baron, I understand the implications of doing filesystem backups while MySQL is running. I was referring to rsync’s done with MySQL shutdown. EX, when I’m trying to setup a new slave. I’ll shutdown the master, rsync off the data dir, (incl, ~30G ibdata1 file) spin up the master, spin up the slave, attach the slave to the master, and find out (usually within an hour or so) that some queries are failing on the slave because of some kind of corruption in the innodb tables. I’ve seen this behavior with several versions of rsync, with several versions of mysql, on several different linux distros, and on many different pieces of hardware. When I use scp, it works fine. When I use rsync, it “reliably” fails. My theory is that there’s something going on in rsync when the file gets over 4GB, but that’s a very loose theory.

With every new version of rsync I get excited and try my test case again, but sooner or later I think i’m going to have to put together a bug report for the rsync guys.

By: Baron Schwartz

Baron Schwartz — Fri, 25 Dec 2009 09:27:30 +0000

Kattare, nothing is wrong with that, if you can do that. Lots of people can’t just take a slave offline like that, though.

You CANNOT rsync InnoDB’s data with the server running, period. This is a common mistake. InnoDB’s background threads are busily writing data, even if no queries are being executed. If you wait long enough all the pending work will get done, but that’ll take an indeterminate amount of time. Just shut down first.

By: Kattare

Kattare — Thu, 24 Dec 2009 04:53:33 +0000

Is there anything wrong with doing a slave stop / run the backup (mysqldump & filesystem both) / slave start procedure on a backup slave?

We lean in that direction whenever possible and I was surprised to not see it mentioned here anywhere…

Has anyone else noticed that rsync’d ibdata files don’t seem to maintain their integrity whereas an scp’d one does? Maybe rsync just doesn’t handle them well when they get over a certain size? I can’t seem to figure it out.

By: Vitaly

Vitaly — Thu, 02 Jul 2009 12:33:26 +0000

Thank you!
In fact, I’m migrating my backup scripts to another host now.

By: dermoth

dermoth — Thu, 02 Jul 2009 11:43:18 +0000

Vitaly,

That may be a bug…. In 2.6.22 I had one where running I/O-intensive applications in the IDLE class would slow down everything (both MySQL and the IDLE application were running much slower than when they were both in the default class).

Check the crond scripts or use ionice to verify you’re not using classes already (i.e. the system may already use the IDLE class for I/O-intensive jobs)… Other than the my only suggestion would be to upgrade.

By: Vitaly

Vitaly — Thu, 02 Jul 2009 07:46:54 +0000

Thank you, I read your post, probably I just didn’t understand it
First, I’m still with RHEL4 (2.6.9-42.0.3.ELsmp) – it means, I cannot switch between schedulers.
Second – my question is mostly why slowing down is *so* drastic, in many cases it seems like mysql is stucked at all.

Thank you!

By: dermoth

dermoth — Wed, 01 Jul 2009 16:29:24 +0000

Vitaly, you should read my post above… There’s pretty much the answer to your question there.

Nice value affects only processor scheduler and has no effect on IO-intensive tasks. IO classes (ionice) works only with CFQ, and as I point out in that post, even when using IO classes to give priority to MySQL, CFQ makes MySQL run slower than Deadline on my hardware.

I haven’t tested with the Anticipatory scheduler, however it should be even worse by design; Anticipatory should make Mysql wait even more on IO expecting sequential reads; while it will likely make random requests.

Once you find the fastest scheduler for your load/hardware, you may want to tune the scheduler parameters to see if you can get any better results. These are in /sys/block//queue/iosched/.

By: Vitaly

Vitaly — Tue, 30 Jun 2009 09:11:26 +0000

Thanks for the interesting article!
I use my DB server as backup/archiving server for some data (no MySQL related).
I understand that MySQL will respond slowly during when some IO-demanding processes are running (gzip, etc…).
But many times my MySQL DB server just isn’t responding – at least, the same query that usually takes about 2 secs, isn’t responding after few *minutes*!
I played with “ionice for poors” perl script and at the end I’ll probably move all non-DB thins to another box, but I’m just curious – is this some bug in scheduler? in MySQL? or this is expected behaviour?