Comments on: Disaster: LVM Performance in Snapshot Mode

By: hawaii

hawaii — Thu, 24 Feb 2011 17:43:43 +0000

Hi Peter,
the mk-table-checksum tool is something I already have in place in my replication concept. So good to see I’m on the right way there.
I will take a look on xtrabackup as well and will compare it to the LVM solution. Restore time is the most important thing so far.

Thank you all!

By: Peter Zaitsev

Peter Zaitsev — Thu, 24 Feb 2011 16:50:25 +0000

Hawaii,

Using Slave for backups is common solution. One thing to mind is you better ensure you’re checking your replication is in sync with mk-table-checksum or other tools. There are many reasons replication may go silently our of sync and you do not want to discover that just after restore.

If you’re using Innodb also check out xtrabackup it has lower overhead than LVM and it also has throttling you can use to limit the impact it has on production.

By: hawaii

hawaii — Thu, 24 Feb 2011 16:36:43 +0000

Thank you LenZ. That’s what I have already in place. I kust wanted to confirm LVM snapshot backup is still the way to go for large scale mysql databases.

By: LenZ

LenZ — Thu, 24 Feb 2011 16:20:21 +0000

@hawaii: If you can’t afford the performance impact (which many users can’t), you’re probably better off by setting up a replication slave for backup purposes (it can of course be used for other things as well). This gives you much more flexibility, since the choice of the backup method is no longer restricted by any limitations you might face on the master server. You could even take the replication slave down for the time of the backup, it would automatically resync/catch up after you turn it on again.

By: hawaii

hawaii — Thu, 24 Feb 2011 16:01:12 +0000

I am looking for a proper solution to create backups of a mysql database with around 150G of data. After some reading LVM snapshots was my first choice as I can’t afford longer downtime. Also logical backups with mysqldump are not an option as restore time would be just insane (although I have them for worst case scenario).

While reading across the topic I stumbled over this thread, so for now I’m quite sceptic.
Once the snapshot would be taken, the DB would need to go right into production again so I can’t afford any performance loss while taking the backup of the snapshot.
So I guess the only thing helping me out would be something like a backup slave just dedicated for backups?
All other solutions don’t seem to be a go for larger scale DB’s or am I missing something?

I would be thankful for a head-up on this.

Best

By: LenZ

LenZ — Mon, 01 Nov 2010 11:18:25 +0000

Brett: Sure, if you move the hot spots of activity to separate volumes, there will be less amount of data that has to be copied around while the snapshot exists. It also helps to remove the size required to keep a snapshot active. But it also means that you won’t be able to create a consistent/atomic snapshot across all your data. If that’s not a concern, it sounds like an option to look into.

By: Brett

Brett — Mon, 01 Nov 2010 07:32:53 +0000

I know many of you here are talking about Databases setups etc, but on more generally loaded systems, webservers, email, etc would breaking down the size of Logical volumes relieve some of the problem? If you took a snapshot of an extremely large and active Logical Volume then the amount of IO changes would obviously be large and hence overall performance would drop during snapshots. But if you broke down your drives into many smaller Logical Volumes then the performance hit per snapshot (per logical volume) would be less overall. I am talking about more general purpose setups, not pure database systems. This is what I am currently looking at now as a possible way to relieve the snapshot “blues” for our servers. Rather than having one large logical volume I am considering breaking them down into many smaller logical volumes. Do people think this could be effective in reducing the performance hit of snapshots?

By: peter

peter — Sat, 16 Oct 2010 05:39:42 +0000

Mirrors,

mysql-bench is very poor tool for performance estimation for majority of workloads. It is using very small data set and in this case most of the tests will not be IO bound. test-create calls fsync() for every table created which is likely why it is affected. Test with your real workload or similar on data set of your scale to get meaningful results.

By: Mirrors

Mirrors — Thu, 14 Oct 2010 14:36:03 +0000

Peter,
I tested mysql-bench suite,including test-create,test-insert,test-select,but only test-create suffered from LVM snapshot,and about 10 times slower.But it’s seems that select/insert operations are not affected seriously.Is that right?

By: peter

peter — Wed, 04 Aug 2010 02:11:01 +0000

Andreas,

Yes… The performance when you copy from snapshot often just drops to the floor. You can check with different IO scheduler but in general I do not know perfect solution.
Consider Xtrabackup – we use it in many cases to avoid having penalty of LVM snapshot