Great blog – I enjoyed reading it. BTW, I am following up on an old posting of yours “How much overhead DRDB could cause?”.

You have mentioned in the blog and I quote…
……….
“If you’re looking for less overhead for DRBD with fast storage (ie BBU) you should be looking at low latency network communications.

It would be actually rather interesting to see DRBD to have direct support for Infiniband or Dolphin interconnect sockets which are getting low cost these days and which could offer significant performance improvements for DRBD. Though you should be already be able to use these using standard TCP/IP communication, which already makes things a lot faster than 1Gb Ethernet.
Though this is only a theory – I have not had a chance to play with DRBD on this kind of networks yet.
”
……….

I would like to post an update that we have used DRBD with Dolphin Interconnect/Fast Storage (using our new PCI Express based flash storage product called StorExpress). The combination of Dolphin’s StorExpress and its low latency/high bandwidth communication model using Dolphin Express interconnect/SuperSockets software provides an ideal environment for supporting an extremely efficient replicated block storage mechanism. As a result, we have seen around 50% improvement in performance over a 10 Gigabit Ethernet solution for a wide ranging set of block sizes. Dolphin Interconnect along with StorExpress based storage provides the best replication performance for DRBD.

For more details, please see
http://www.dolphinics.com/products/storexpress.html
http://www.dolphinics.com/solutions/dolphin_express_drbd_speedup_linbit.html

Thanks.

I do have some experience of running DRBD over Infiniband. We used 3rd Gen Mellanox PCI-E 20Gb Infiniband in conjuntion with IPOIB-CM with a 32Kb MTU (Useful for other use, not DRBD). I get total synch rate of around 300Mbytes/sec with multiple arrays synching (This is limited by the arrays, tcp/ip performance manages about 700Mbytes/sec). Its not really bandwitch im interested in though, its random write performance which seems to be affected somewhat by DRBD.

RDMA (Used by Infiniband and IWarp NICs) would likey greatly help reduce additional DRBD write latency (More important to most people) and increase usable bandwidth to about 1400Mbyte/second in each direction on current 4th gen 20Gb Infiniband cards.

I was told by Linbit 6 months ago that RDMA support was only on the maybe list for DRBD 9 and support for RDMA was quite a long way away. Have you heard any additional info on RDMA support in DRBD?

Indeed I write about latency – because this is where the problem frequently would be with MySQL running on DRBD.
Good to see we’re close to the agreement on this

Regarding – you would not notice unless application is write intensive – sure. It is worth to highlight again DRBD overhead comes from Writes only. Reads are served from local drive and have little overhead.

it is, as always, important to point out that “performance” in I/O is a combination of two distinct things that may affect applications in very different ways: throughput and latency. Your post talks mostly about latency, so that’s what I am going to tackle here.

Consider a local disk with about 3ms latency for a small (one sector, 512 bytes) write request. Consider further that with Gigabit Ethernet, we get about .1 to .3ms network latency. DRBD’s latency penalty amounts to roughly twice the network latency, so that would be .2 to .6ms or about 7 to 20% latency overhead.

Now when you introduce a RAID controller with battery-backed write cache to the setup (and you should), your local I/O latency decreases dramatically. Now your local writes “complete” in something like .1 to .3ms. So in relation, all of a sudden DRBD’s latency penalty “jumps up” (in percentage, not in absolute terms) to perhaps 100% or 300%. And there’s little you can do about this, because it’s almost impossible to tune the Gigabit Ethernet-based TCP/IP stack to deliver packets significantly faster.

It should be noted that unless your application is highly write intensive, you’ll probably not notice much of this. However, for extremely high write loads, DRBD currently does add comparatively large performance penalties percentage-wise. Which is subject to change when we move off Ethernet-based IP as our sole network protocol. Some of the low-latency network technologies you mention are already being worked on — stay tuned on the drbd-user mailing list for announcements.

And just in case other readers here are interested in learning more about DRBD performance tuning considerations, I encourage you to watch my blog (http://blogs.linbit.com/florian) for a free DRBD performance tuning webinar we will be announcing shortly.

Cheers,
Florian