The problem could be repeated with SysBench as follows:

On Dell R900 with CentOS 5.2 and ext3 filesystem we get 1060 transactions/sec with single thread and sync_binlog=1 while with 2 threads it drops to 610 transactions/sec Because of synchronous serialized writes to binlog we're not expecting significant improvements in transaction rate (as the load is binlog fsync bound) but we surely do not expect it to drop ether.

To ensure this is not RAID controller related we repeated the run on the in memory block device and got very similar results - 2350 transactions/sec with 2 threads with sync-binlog=0 and just 750 transactions/sec with sync-binlog=1

We tried running data=writeback and data=journal just to make sure but results were basically the same.

Using XFS instead of EXT3 gives expected results - we get 2350 transactions/sec with sync_binlog=1 and 2550 transactions/sec with sync-binlog=0 which is about 10% overhead.

EXT2 filesystem also behaves similar to XFS so it seems to be EXT3 specific performance issue. We do not know if it is CentOS/RHEL specific and if it is fixed in the recent kernels. If somebody can run the same test with different kernels it would be interesting to know results.

You also may wonder how could this problem be unnoticed for a while ? First it only applies to the binary log. Innodb transactional log does not have the same problem on EXT3. I expect this is because Innodb log is pre-allocated so there is not need to modify meta data with each write while binary log is written as growing file. Another possible reason is small sizes of writes which are being fsync'ed with binary log. Though these are just my speculations - we did not investigate it in details.

Another reason is - it only happens with high transaction commit rate. If you're just running couple of hundreds of durable commits a second you may not notice this problem.

Third - sync-binlog=1 is an option great for safety but it is surely not the most commonly used one. The web applications which often have highest transaction rates typically do not have so strong durability requirements to run this option.

As the call for action - I would surely like someone to see if EXT3 can be fixed in this regard as it is by far the most common file system for Linux. It is also worth to investigate if something can be done on MySQL side - may be opening binlog with O_DSYNC flag if sync_binlog=1 instead of using fsync will help ? Or may be binlog pre-allocation would be good solution.

Entry posted by peter | 17 comments

Add to: | | | |

Today I'll look at the same topic from the consultants view. When consultant should suggest hardware upgrade and when it is not in a simple checklist form.

How good is hardware ? Sometimes people use so crappy hardware it would be much cheaper for them to upgrade before purchasing much of professional services. Though in some cases people like their system to be optimal and so they want to run it on some old box even if it costs them more to optimize it. It may be valid choice allowing to take a hardware boost later down the road when you need a major performance increase and do not have a time for big application changes.

Is current hardware usage balanced ? Say you have 10 servers one of them is overloaded and 8 are almost idle. In this case proper balancing is the answer rather than getting more hardware. Of course not all of the applications can be easily balanced but it is also surprising how many cases of people with reasonable sharded or replicated architecture suffer from wrong balance. Balancing can be simple operations act or require application changes which is another variable.

Are there any spikes in hardware use ? Often performance problems happen only Sometimes..... for example nightly when you do a backup or at 15 minutes off hour when you have some cron job running. In such cases evening out the usage is often better choice than hardware upgrade and it also can be done easily.

Is MySQL, Queries, Architecture optimized well enough ? Do not advice hardware upgrade as the first thing when you can triple performance by simple my.cnf change. Well enough is a tricky term though. You need to balance things and see what can be easily done by consultant or the customer and what is not. Adding the indexes is easy. Changing the query in your own application is easy but is hard for third party application, especially if it is close source. Big schema changes, caching, sharding can be even more complication - it depends. The bigger your application is going to be the more optimal you want to be on application level to be efficient.

What is exactly your goal with hardware upgrade ? Same as with software optimizations and changes you do should have a goal. You add the index to avoid full table scan and make given query faster. You add memory to avoid disk IO and make lookups faster. The goal in the application performance - making throughput better, query faster should lead you to goals in hardware (faster random IO, better caching, faster execution) which you can use to understand what needs to be done (gettings SSD, upgrading memory, upgrading CPUs). There are number of balancing questions you have to solve too such as SSD may not need so good in memory caching any more etc. There is no such thing as better hardware really, just hardware which serves application needs better. Many of us probably remember as moving from 4 Core systems to 8 Core reduced performance for many MySQL systems even though hardware itself was superior.

There are surely more things you can look at but these are simple obvious things you can keep in mind. Do you think I should add something else ?

Entry posted by peter | 5 comments

Add to: | | | |

I though wow did they really make it work, considering how many bottlenecks remain in MySQL.

What article really tells us ?

First the number 256 - this is not number of Cores... this is number of hardware threads which is not exactly the same thing. Each T2 Plus CPU has 8 cores, which with 8 threads each giving 64 threads per chip or 256 threads all together.

Now what is about MySQL scaling to use these 32 cores with 256 threads ? Especially with the goal of "Do it with minimal tuning i.e as close as possible as out-of-the-box" ? Do we simply start MySQL server and change couple of defaults to make it work ?

No! To get really good performance we have to setup 32! MySQL instances on this box (approximately one per core).

Comparing single instance performance (4350 statements/sec) to the total performance archived with 32 instances (79334 statements/sec) we can see the single instance gets 1/18 performance archiveable on the system which is of course extremely poor number.

The scale out and ability to scale by using as many MySQL instances as you like is a very good application architecture but unless you have everything absolutely automated in the Google way I do not think you would enjoy running 32 MySQL instances on the box for the single application just to get decent speed.

Now I should point back to my old post about T2000 Perfrormance with MySQL The Matt Ingenthron commented “Sun” is not, to my knowledge, “aggressively pushing T2000 as Scalable MySQL Platforms”. - Well what this one would be ? Of course it is not T2000 any more and the CPU performance got a bit better but as I understand single thread performance is still many time slower than one of recent Xeons which makes it really tough call for MySQL which can't perform many operations in parallel.

Now you might got a feeling 79000 queries is a lot ? It is hard to tell without knowing what the queries are but if you just want to look at the queries - check this out - these are a year and a half old benchmarks on the single MySQL instance showing over 50.000 queries on 8 cores (and close to 40.000 queries/sec on 4 slow AMD cores).

Do not get me wrong. The multi-threading architecture Sun has is great for many applications... however it is NOT great for MySQL unless you really really do not care about single thread performance (REPLICATION, ALTER TABLE etc) and willing to run insane number of MySQL instances per server.

Entry posted by peter | 13 comments

Add to: | | | |

So am I scale-up advocate ? No. But it is quite frequently systems are designed to have "working set to fit in memory" to perform well and such systems can avoid good consolidation factor for such application, or would allow them to delay sharding.

This box also has 4 CPU sockets which means 16 fast cores and 128GB of memory becomes commodity - Quite a challenge for MySQL to take

I have not had a chance to play with such box myself besides couple of customer production installations but it looks pretty sweet.

Entry posted by peter | 13 comments

Add to: | | | |

Most hosting providers -- even the big names -- continue to install 32-bit GNU/Linux operating systems on 64-bit hardware. This is a serious mistake.

You have to tell them to install a 64-bit operating system. If you don't then you will come to a point where your needs grow and you want to use more memory -- and they will gladly install 8 or 16GB of memory for you, but MySQL can't use it because it runs in a single process, which is limited to about 2.5GB of memory. And then you have to rebuild the whole operating system from scratch. But you don't want any downtime, so you have to buy another server, set it up as a slave, switch your site to use it, and then rebuild the old server. That 32-bit OS turned into a pretty expensive mistake.

I do not know why the hosting providers keep doing this. Just yesterday I got a quote from a hosting provider for a medium-high-end system with 8GB of RAM, and forgot to tell them 64-bit OS, and they actually listed 32-bit explicitly on the quote -- useless! I would estimate about half of all the hosted systems I've seen so far have this mismatch. I don't know why they do this -- maybe there is a reason, but I don't know it and it looks pretty silly to me. 64-bit hardware and operating systems aren't new anymore. In fact, 32-bit is hard to find in server-class hardware these days. So it certainly looks like the hosting companies need to change what they're doing, but maybe there's a different reason.

Entry posted by Baron Schwartz | 30 comments

Add to: | | | |

But thing a lot of people miss is - Being Scalable is Not Enough - you need to scale from reasonable base to claim the good performance, and this is where T2000 performs subpar in many cases.

I often hear about people complaining queries take much longer on T2000 compared to recent Intel or AMD CPUs when there is no concurrent load - It is reported T2000 can be as much as 5-15 times slower in this case depending on the workload.

Here is example run of purely CPU consuming "Benchmark" function for 2.6Ghz Intel Xeon vs T2000:

As you can see this is hell a lot of difference !

Depending on your application performance with single thread may be important or non important for you - it is surely important for the slave if you're having active replication, if you're running time sensitive long running CPU bound queries or if queries contribute significant time to generating web page.

For example if on Xeon queries take 50ms to generate the page, the MySQL Latency you may see on T2000 may be as high as 500ms which would be well above performance guidelines for many web applications.

I'm hearing Sun is working on new CPUs which would offer significantly higher single thread performance, but at this time I have to be very careful advising this platform to the customers.

Entry posted by peter | 12 comments

Add to: | | | |

I helped Violin Memory to optimize MySQL for customer workload and Violin memory and also had a chance to do some benchmarks on my own. 2*Quad Core Xeon running CentOS5 was tested using ext2 filesystem and SysBench tool.

Using 16K read sizes (matches Innodb page size) I could get 1.2GB/sec (80K req/sec) for reads and about 500MB/sec writes with 16 concurrent threads. Things scaled well and with 256 threads I got even a bit better performance.

Interesting enough utilization in iostat never went over few percents and load was mostly CPU bound.

Next I went on testing MySQL. My goal was to simulate as much IO as possible so I use Sysbench to get 300Mil rows table and ran primary key lookups with uniform distribution some 90% of which would need to do IO to get their data from the disk.

With Innodb results were quite poor - I got about 6500 req/sec from one thread which grew to 13000 req/sec when 8 concurrent threads were used. Compare this to 20.000 queries/sec from single thread if all data fits to bugger pool, scaling to over 80.000 queries/sec with multiple threads.

So Innodb took very high hit supplied with high performance IO subsystem. But not only performance was poor with single thread it also did not scale well with increased concurrency even though we had 8 cores available for disposal.

Unfortunately oprofile did not work on the box in question so I could not investigate where exactly CPU is wasted. My first obvious quess was Innodb checksumming (innodb_checksums=0) and indeed that allowed to get to 9000 queries/sec from single thread, however it still peaked out at 13.500 queries/sec with multiple threads which corresponds to even worse scalability. Disabling adaptive hash index could take us to 14.500 req/sec which was also a bit surprising as we're speaking about read-only workload so it is quite surprising adaptive hash index actually hurts performance in this case.

My guess is Innodb just was not really designed and tested in such condition - normal case is to allocate cache memory to buffer pool so IOs will mostly come from drives directly which means hundreds or may be thousands of IOs per system for the whole system which allows to waste some CPU cycles handling them without taking large performance hit.

I really hope this would be one of the Items besides CPU scaling which Innodb team will put on their roadmap. With SSD coming this will be important. And it is also very easy to repeat and test - just run Innodb from the RAM drive

Next I tested MyISAM which is well known for its simplicity and so fast execution path from getting data from the drive and sending it back to the client.

I tested MyISAM with myisam_use_mmap option which did not seems to give any significant performance benefit, may be even reducing performance few percents. I also tried running with key buffer on and off. Interesting enough disabling key buffer actually worked better for this workload avoiding to avoid contention.

In the best conditions I got about 13.5K queries/sec from MyISAM (double of Innodb already) which scaled to 70.000 queries/sec peaking out at 16 threads.

However even with MyISAM we got CPU bound before we could reach the system capacity - these 70K queries/sec generated just over 50K IOs/sec while capacity was over 100K IOs/sec (more than with Innodb as these are smaller sized)

These results reiterate one important things about Innodb - it just loves to have as much data in buffer pool as possible for best performance.

Hopefully some time in the future I get a chance to do more testing with this nice hardware and may be check out other storage engines and other workloads or get some profiling results.

Entry posted by peter | 21 comments

Add to: | | | |

Here is what we're looking for in particular:

- More than Dual Core Opteron systems. Would be good to see how they scale
- More than 8 core systems
- Non x86 based systems (Niagara etc)
- Fancy IO subsystem - more than 8 hard drives
- SSD based storage.

If you have something of this sort available let me know.

Entry posted by peter | 10 comments

Add to: | | | |

What I usually look for MySQL is performance in random reads and random writes. Sequential reads and writes are rarely the problem for OLTP workloads, so we will not look at them.

I also prefer to look at performance with O_DIRECT flag set to bypass OS cache. This may execute separate code path in kernel and so has a bit different performance pattern compared to buffered IO (even followed by fsync regularly) , but it allows to easily bypass OS cache both for reads and for writes and so does not require creating large working sets for boxes with significant amounts of memory (or reducing amount of usable memory).

The system I'm testing this on has 256MB BBU (Battery Backed up Cache) on RAID controller so we will test two workloads size - first one is small which fits in the cache and the second one will be large enough so it does not. This allows us to see both by RAID cache and uncached IO performance.

We're interested in cached reads because they may show latency to the RAID cache if RAID read cache is enabled. If it is disabled you may be reading from Drives cache or even drives themselves which will affect performance significantly. In any case especially playing with data size a bit you will well learn how your cache behaves.

Were even more so interested about writes which show us how many cached writes per second we can do - this is important for log writes, which are synchronous IO operation database performance which typically have rather close data locality.

We're also interested in uncached reads and writes because this correspond to general database workload.

It is worth to run the test with 1 thread and with some higher number (say 64) to see how things scale.

The tool I'm using for this is SysBench which was designed by my team when I still worked for MySQL and we specially implemented bunch of tests to ease hardware evaluation for things which are important for MySQL.

To prepare small 128MB single file working set we can use the following command:

And when we can run the test:

The output is pretty verbose so it is quite scary for many people, however there is basically one number important here 12897.22 req/sec - so the RAID cache on this drive can do about 13.000 16K reads/sec from the cache, not bad at all.

With 64 threads I get 23727.46 req/sec which shows even in such cached mode you can get better performance by having many outstanding requests.

Running with more than one threads you may also take a look at another table SysBench displays:

So we get about 3ms 95 percentile request time - not bad - all request were executed with pretty uniform performance.

Let us now do the test with single thread and write-through RAID cache (as if there would not be any BBU)

Auch... We get 142.62 Requests/sec which is extremely poor considering this is 8 drives array. You would likely get a bit better with sequential log writes but it is not going to be much higher.

Lets see what we get with 64 threads... hm again 142.83 Requests/sec and looking at VMSTAT I can see only 1 blocked process all the time which does not sounds right.

Honestly I did not expect this result while running benchmarks for this articles (and this is always very exciting to find something unexpected while doing Benchmarks). Fortunately I've already seen this some time ago though I hoped this issue is long fixed since that time... Though this box is running CentOS4 which is not the newest OS out where.

The problem is basically in case of O_DIRECT writes there is only one IO can be happening per file at any given time. To show this is the problem let us create working set consisting out of number of files and repeat the run:

Now we get 827.94 Requests/sec requests per second, which is not bad considering this is RAID5 volume.

Let us now repeat the tests for WriteBack cache configuration:

We get 4735.27 Requests/sec from 1 thread and single file; 4740.19 Requests/sec from 64 threads and 1 file and 5228.68 Requests/sec with 64 threads and 128 files.

As you can see numbers are a lot better especially when there is single thread doing IO (just as with database log). We can see serialization with O_DIRECT however the impact is much less in this case compared to Write Through mode because we're mostly limited by RAID controller capacity.

Let us now do test with larger test set. We'll use 2GB in 128 files (because we already discovered synchronization issue with O_DIRECT). This sounds small but as controller cache is just 256M it should be good enough.

There is also one trick running such larger scale benchmarks. SysBench by default has repeatable sequence of requests, because it does not initialize random generator. If you're doing multiple runs make sure to add --init-rng=1 option otherwise results can be skewed.

So we do run as:

For single thread we get 259.28 Requests/sec which approximately matches what single drive can do. This is important lesson - even if you have many hard drives you can't really get advantage of them unless you can get multiple outstanding requests.

Performance with 64 threads was 2334.63 Requests/sec which is perfect scalability to 6 hard drive this array has. We have even a bit more than 8x because 64 outstanding requests allow RAID controller to optimize seeks.

Lets us see how writes are doing:

For writes we're getting 583.43 Requests/sec for 1 thread and 605.88 Requests/sec for 64 threads.

As you can see we got more writes for single thread than we had reads and also using multiple threads for writes did not improve performance as dramatically as for reads. This is because as we have BBU writes are just stored in controller cache and flushed to the disk in background - in this case there is little difference if we use one thread or multiple threads.

As you also can see BBU is helpful even for random writes if they are performed from single thread - so it does not only help database log writes but has much larger positive impact on workload.

I should also note it is very good idea to watch VMSTAT while doing the test. Here is the snipped from good VMSTAT from the read test:

As you can see all 64 threads are waiting for IO. Now lest see our write test which seems to have some serialization issues

You can get similar information by analyzing iostat numbers and that well may be more accurate. Though looking at VMSTAT is often good enough to see the problem.

P.S If you're wondering which hardware I used for benchmarks - it is Dell PowerEdge 2950 with PERC5 and 6 10K RPM hard drives in RAID5.

Entry posted by peter | 18 comments

Add to: | | | |

This is also almost the case with Hard Drives - there are possible performance differences based on where data is stored on the disk, amount of remapped sectors etc. There is also database and file system fragmentation however these also tend to accumulate in predictable fashion.

If you have RAID controller this well may not be the case - to protect your data RAID controller may implement bunch of algorithms which can affect performance dramatically.

For example speaking about PERC5 (LSI MegaRaid) - Pretty typical controller from Dell installed on PowerEdge 1950, 2950 etc you should be aware of couple of things

Battery Learning and Charging Build in Battery has to pass through learning cycle every 3 months or so and this cycle takes about 7 hours according to the docs. During this time battery backed up cache will be disabled and system will operate with write through cache possibly slowing down write performance several times.

Patrol Read This is feature which should discover bad sectors before it is too late and it is doing so by doing disk read checks periodically. When it wakes up it will some IO resources (30% by default) which will affect your performance some way.

Consistency Checks This is another thing which I've seen initiated by controller (though I'm not sure on this one) - which pretty much checks the disks are in sync - this also can slow down performance dramatically.

So what you can do about these ?

First you should not have any of this to come as surprise for you when you discovered your server just stopped performance during the time you planned investor show case or other important event. Learn what cron jobs does your RAID card have and see how they can be controlled - may be schedule them during least busy intervals or something similar.

Also you should be ready for degraded and rebuild RAID mode - when one of the disk fails and you replace it with another one which needs to be rebuilt. This means you already should leave some slack of the system. It often would be enough for consistency check and patrol read but not for battery backed up cache being temporary disabled.

Another thing you can do is of course switch to another server and take this down for maintenance if this learning process can't be scheduled when it is non intrusive. To do this properly however you need to know when it is about to happen.

Entry posted by peter | 6 comments

Add to: | | | |