Part 1 of our series set-up our “test” application and looked at boosting performance of the application by buffer MySQL with memcached. Our test application is simple and requires only 3 basic operations per transaction 2 reads and 1 write. Using memcached combined with MySQL we ended up nearly getting a 10X performance boost from the application. Now we are going to look at what we could achieve if we did not have to write to the database at all. So let’s look at what happens if we push everything including writes into memcached.
Wow that’s shockingly fast isn’t it! I guess being completely in memory helps for this app. What is very interesting is accessing 100% of the data in memcached gives very similar numbers to accessing 100% of the data in memory in the DB ( part 1 benchmarked a 4GB bp as being able to handle 7K TPS)… something is not 100% right here. It stands to reason that memcached should be faster for this application then the DB. Its just doing two gets via key and 1 set. So why the similar numbers?
Well glad you asked. It’s the API. The api in this case was Cache::Memcached, by switching to using Cache::Memcached::Fast look what happens:
That is a nice jump in performance!
Using Memcached::Fast was kind of a mixed bag when looking at the benchmarks for mixing MySQL and Memcached in my tests:
In this case I think the Fast api was slower when working with MySQL with a 256m BP because the slower returns from memcached acted as a bottleneck to thin the demands on MySQL to write data, smoothing out the work load. When we eliminate this bottleneck with the Fast api, MySQL gets hammered. This type of thing happens a lot. For example an application is CPU bound, so you add more processing power, but then you hit disks harder and now your disk bound.
A couple of good things to remember here: #1 resolving 1 bottleneck can open another bottleneck that is much worse. #2 is to understand that not all API’s are created equal. Additionally the configuration and setup that works well on one system may not work well on another. Because of this people often leave lots of performance on the table. Don’t just trust that your current API or config is optimal, test and make sure it fits your application.
So adding Memcached on top of MySQL for our test application can significantly boost performance. But you notice that if we were running 100% in memcached and could cut out MySQL we could get 2.5x more performance over a mixed solution and 100X over just stock MySQL. As the number of writes against the database increase this gap will increase. So let’s ditch the database! But wait! you need the DB for persistence, right?
It depends. A database may not be the best fit for every application. There are several “NOSQLâ€Â solutions out in the open source space that can give you some of the ease of a Memcached but with persistence most people use their database for.  Each application is different and understanding the application’s requirements is key to picking an appropriate solution.  I am going to look at several database alternatives over the next few months. I need to start somewhere, so I decided to start with Tokyo Tyrant and Cabinet.   So stop in next time for part 3 of this series where we will focus on running the same tests against Tokyo Tyrant.
Wow that’s shockingly fast isn’t it! I guess being completely in memory helps for this app. What is very interesting is accessing 100% of the data in memcached gives very similar numbers to accessing 100% of the data in memory in the DB… something is not 100% right here. It stands to reason that memcached should be faster for this application then the DB, two gets via key and 1 set. So why the similar numbers?
Well glad you asked. It’s the API. The api in this case was Cache::Memcached, by switching to using Cache::Memcached::Fast look what happens:
That is a nice jump in performance!
Using Memcached::Fast was kind of a mixed bag when looking at the benchmarks for mixing MySQL and Memcached in my tests:
In this case I think Fast was slower when working with MySQL with a 256m BP because the slower returns from memcached acted as a bottleneck to thin the demands on MySQL to write data, smoothing out the work load. When we eliminate this bottleneck with the Fast api, MySQL gets hammered.
A couple of good things to remember here: #1 resolving 1 bottleneck can open another bottleneck that is much worse. #2 is to understand that not all API’s are created equal. Additionally the configuration and setup that works well on one system may not work well on another. Because of this people often leave lots of performance on the table. Don’t just trust that your current API or config is optimal, test and make sure it fits your application.
So adding Memcached on top of MySQL for our test application can significantly boost performance. But you notice that if we were running 100% in memcached and could cut out MySQL we could get 2.5x more performance. As the number of writes against the database increase this gap will increase. So let’s ditch the database! But wait! you need the DB for persistence, right?
It depends. A database may not be the best fit for every application. There are several “NOSQL†solutions out in the open source space that can give you some of the ease of a Memcached but with persistence most people use their database for. Each application is different and understanding the application’s requirements is key to picking an appropriate solution. I am going to look at several database alternatives over the next few months. I need to start somewhere, so I decided to start with Tokyo Tyrant and Cabinet.
While the figures are good, it’s important to point out that the MySQL solution provides durability (i.e. no data loss while memcached does not.
I’m looking forward to the results of a key value (KV) product that does provide persistence storage. Are you going to be looking at others like Redis/Project Voldermot as well?
Just wanted to let you know that the link to part 1 is broken. Great series, I look forward to the “NOSQL” posts!
Hmm failed the challenge, 5 + 0 doesn’t equal 5 apparently. Let’s see if I can get 0 + 10 correct…
Matt,
I’m not sure I quite follow the 100x gain from “stock” MySQL
What exactly are you comparing here ? Is it apples to apples ? Or are you comparing the case when everything fits in memcache but only small portion fits in MySQL.
Also you show 3x difference between memcache and memcache fast – what is the difference here ?
Also are you running tests locally or over network ?
@Peter,
The 100x claim is a bit misleading, as memcached has the ability to cache everything -vs- MySQL which I did handicap. The idea was to compare a memory footprint in MySQL -vs- the same memory footprint in Tokyo Tyrant ( Tyrant only had 256M allocated to it as well )… but memcached has no disk based component so everything has to be in memory… however this can be distributed over multiple machines.
Memcached::Fast is implemented in C.
I have run both, but I figured locally would be best for these numbers… so everything here is local access.
Is that Memcached::Fast from there: http://search.cpan.org/dist/Cache-Memcached-Fast/ ? Or something else?
Great series Matt.
I’m looking forward to read about the performance with the so-called nosql solutions, specially Redis and Voldemort. By the way, the link to part 1 is broken.
I’m looking forward to see a Redis versus Memcached test. Redis is persistant and has replication feature.