In this case we’re just using one snapshot (which is enough for database backup) and it does not work.

I have not tried Zumastor – if you can rerun my benchmarks and post results that would be quite interesting.

Have you tried Zumastor snapshot and replication tools?

Zumastor is an enterprise storage server for Linux. It keeps all snapshots for a particular volume in a common snapshot store, and shares blocks the way one would expect. Thus making a change to one block in a file in the original volume only uses one block in the snapshot store no matter how many snapshots you have (verbatim from Zumastor how-to).

LVM snapshots design has the surprising property that every block you change on the original volume consumes one block for each snapshot. The resulting speed and space penalty makes the use of more than 1 or 2 snapshots at a time impractical (verbatim from Zumastor how-to page as well).

Yeah I will try it again when I have the chance…. also will see with XFS.

The problem is based on discussion we’re still looking at 3X of IOs (one read and 2 writes) which is already too large.

In this case it was default. If you suggest some parameters which should help performance let me know we should try it.

Thanks for posting results.

[root@udbqa006.sf2p /home/pv/work/src/randomio-1.3]# ./randomio /dev/mapper/VolGroup20-mysqldata 100 0.2 0 4096 60
total | read: latency (ms) | write: latency (ms)
iops | iops min avg max sdev | iops min avg max sdev
——–+———————————–+———————————-
1793.9 | 1433.7 0.1 69.6 2853.9 82.4 | 360.2 0.1 0.1 4.2 0.1
1675.6 | 1340.0 0.1 74.6 1596.9 88.0 | 335.6 0.1 0.1 2.2 0.0

[root@udbqa006.sf2p ~]# lvcreate –size 60g –snapshot –name snap /dev/VolGroup20/mysqldata
Logical volume “snap” created

1379.3 | 1104.6 0.1 68.7 1560.2 86.1 | 274.7 0.1 87.0 1555.4 178.1
1061.3 | 848.4 0.1 42.7 544.8 43.8 | 212.9 0.1 300.9 865.7 154.5
1082.1 | 863.4 0.1 43.2 572.5 45.5 | 218.7 0.1 286.7 950.5 145.4
1077.3 | 863.8 0.1 43.0 771.1 45.0 | 213.5 0.1 292.6 795.3 147.4

So that’s our base. Let’s try the same with xfs on a large file -

[root@udbqa006.sf2p ~]# mount
/dev/mapper/VolGroup20-mysqldata on /data type xfs (rw,noatime,allocsize=1g)

[root@udbqa006.sf2p ~]# ls -l /data
total 335544320
-rw-r–r– 1 root root 343597383680 Sep 9 16:05 foo

[root@udbqa006.sf2p /home/pv/work/src/randomio-1.3]# xfs_bmap -l /data/foo
/data/foo:
0: [0..173950911]: 320..173951231 173950912 blocks
1: [173950912..341722991]: 183500864..351272943 167772080 blocks
2: [341722992..509495071]: 367263808..535035887 167772080 blocks
3: [509495072..671088639]: 550502464..712096031 161593568 blocks

[root@udbqa006.sf2p /home/pv/work/src/randomio-1.3]# ./randomio /data/foo 100 0.2 0 4096 60
total | read: latency (ms) | write: latency (ms)
iops | iops min avg max sdev | iops min avg max sdev
——–+———————————–+———————————-
1887.2 | 1508.4 0.1 66.2 1694.0 77.1 | 378.8 0.1 0.1 2.9 0.1
1678.6 | 1341.6 0.1 74.4 1584.6 87.0 | 337.0 0.1 0.1 1.1 0.0
1670.9 | 1334.9 0.1 75.0 1567.0 88.0 | 336.0 0.1 0.1 1.0 0.0

[root@udbqa006.sf2p ~]# lvcreate –size 60g –snapshot –name snap /dev/VolGroup20/mysqldata
Logical volume “snap” created

1592.2 | 1271.3 0.1 74.7 1401.9 89.3 | 320.8 0.1 14.7 1801.1 95.8
1073.0 | 859.6 0.1 43.5 682.1 44.5 | 213.4 0.1 294.2 796.2 150.9
1090.7 | 872.6 0.1 42.9 593.9 44.0 | 218.1 0.1 287.3 717.1 146.0

Good – xfs matches raw – with or without snapshot. Now let’s look at ext3 with the same size file -

[root@udbqa006.sf2p /home/pv/work/src/randomio-1.3]# ./randomio /data/foo 100 0.2 0 4096 60
total | read: latency (ms) | write: latency (ms)
iops | iops min avg max sdev | iops min avg max sdev
——–+———————————–+———————————-
149.2 | 119.3 4.8 668.1 1042.5 54.1 | 30.0 5.2 660.9 1048.3 64.6
164.8 | 131.8 1.8 608.0 740.7 43.6 | 32.9 512.3 603.9 734.5 36.5
181.4 | 145.6 8.0 552.3 687.0 41.5 | 35.8 419.2 547.1 671.7 36.6
197.6 | 157.7 6.8 507.8 1118.2 67.9 | 39.9 0.1 498.9 1078.5 67.3
224.3 | 180.3 4.5 448.2 647.6 45.3 | 43.9 0.1 439.5 623.6 47.1
256.9 | 204.2 5.1 390.9 521.1 42.9 | 52.7 0.2 383.4 505.7 43.1
295.4 | 236.6 3.0 340.5 474.3 44.4 | 58.9 0.1 330.5 461.9 46.4
350.3 | 279.4 5.0 287.5 419.8 43.2 | 70.9 0.1 277.6 401.6 43.8
421.5 | 336.8 1.6 239.7 414.8 44.0 | 84.7 0.1 228.2 394.2 45.6
534.2 | 426.8 3.1 190.0 348.7 41.2 | 107.4 0.1 176.1 310.1 40.5
687.7 | 550.7 2.1 148.4 349.6 39.1 | 137.0 0.1 133.4 282.8 38.9
941.7 | 752.9 1.9 111.2 366.5 43.3 | 188.7 0.1 86.5 243.8 41.3
1258.9 | 1005.9 1.6 87.0 603.9 54.9 | 253.0 0.1 49.3 388.4 45.7
1498.6 | 1200.7 0.1 77.3 1056.2 72.1 | 297.9 0.1 24.1 559.9 46.9
1645.6 | 1313.2 0.1 73.6 1323.9 83.3 | 332.4 0.1 10.0 941.0 39.7
1692.2 | 1354.6 0.1 73.0 1436.4 86.1 | 337.6 0.1 3.2 640.0 26.9
1721.1 | 1377.5 0.1 72.3 1921.1 85.1 | 343.6 0.1 1.2 568.4 18.9
1721.9 | 1379.7 0.1 72.4 1562.9 83.7 | 342.2 0.1 0.2 69.7 2.4
1705.9 | 1364.8 0.1 73.3 1559.8 85.1 | 341.1 0.1 0.1 0.7 0.0
1693.7 | 1357.2 0.1 73.6 1480.5 87.1 | 336.5 0.1 0.1 2.1 0.0
1696.2 | 1355.7 0.1 73.8 1520.0 86.4 | 340.4 0.1 0.1 1.5 0.0

[root@udbqa006.sf2p ~]# lvcreate –size 60g –snapshot –name snap /dev/VolGroup20/mysqldata
Logical volume “snap” created

626.7 | 502.0 0.1 168.7 2581.4 192.8 | 124.7 0.1 121.5 2471.1 203.7
310.3 | 248.7 6.2 321.2 529.4 61.9 | 61.6 7.0 326.0 518.9 62.9
305.5 | 244.0 2.1 326.1 595.8 61.2 | 61.5 10.0 331.3 579.2 61.1
313.3 | 251.2 4.9 319.0 503.5 60.2 | 62.1 12.8 322.3 496.6 59.9
300.5 | 239.7 4.1 331.4 1207.4 81.3 | 60.7 10.0 335.7 1201.1 83.5
312.7 | 250.8 4.1 318.6 587.2 65.7 | 61.8 10.0 324.8 574.1 67.0
309.3 | 247.5 3.3 322.5 540.3 58.9 | 61.8 9.0 326.3 517.9 57.3

What’s going on? I’ve glanced over the code real quick and this is my speculation. There are 2 issues with ext3. The first one is that it is not an extent based filesystem. It is block based. What that means is that there are lots of indirect address blocks addressing your data blocks and they don’t have to be all cached if they are not accessed frequently enough. So a read (or a write) might require a disk read in order to map your logical offset to the actual physical offset in the file. That means extra io’s before you even get to read/write your actual data. xfs (and vxfs and zfs) are extent based filesystems and the block maps of large files can fit into an inode itself if the filesystem is not severely fragmented so mapping your offset is very fast (you can preallocate space in vxfs and in xfs – I use a mount option allocsize=1g).

The second problem is the level of parallelism a filesystem allows when it comes to competing reads/writes. I believe that ext3 is taking an exclusive rwlock on each write and doesn’t release it through the whole io. On the other hand xfs (unless you are growing the file) takes and holds the lock in a shared mode if the file is opened in O_DIRECT mode. This can have big consequences.

Look at the slow start of ext3 – it took nearly 15 minutes to come to the raw speeds. Why is that? Look at the write latencies – at the beginning they are big – which we would not expect with write-back cache. I believe that they are so big because the indirect address blocks were not in the cache. So what should be a very fast write under normal circumstances (write-back cache) turns into a read and a write while holding the exclusive rwlock – blocking all the readers at the same time. Eventually (in 15 mins or so) all of the indirect address blocks are cached and ext3 resumes the “raw” performance.

But once the snapshot is created it plummets again. It’s the same reason – a write turned into a COW read and a write will hold the exclusive rwlock for much longer than if it didn’t have to do a COW operation and so it chokes readers once again. Bigger write ratio has even a more drastic effect on ext3 + snapshot performance.

“randomio” might not be a good representation of what MySQL is doing so we are currently testing xfs with MySQL. But if my speculation are valid then I would expect a smaller degradation with xfs.