However I follow the project SystemTap for couple of years (it was started in 2005), which is supposed to provide similar to DTrace functionality.

Why I am interested in this tool, because there is no simple way under Linux to profile not CPU-bound load (for CPU-bound there is OProfile, see for example
http://mysqlinsights.blogspot.com/2009/08/oprofile-for-io-bound-apps.html). I.e. for IO-bound or for mutex contention problems OProfile is not that useful.

SystemTap is included in RedHat 5 releases, but I was not able to get it running even in CentOS 5.3 (it crashed and hung every so often). Latest updated RedHat 5.4 promised some more fixes to SystemTap, so I decided to give it more try as soon as I got RedHat 5.4 on hands.

Surprising, but now it runs much more stable. I was able to get profiling of kernel and system calls.
Here is simple script to show IO activity per disk per process (well, it is similar to iotop, but iotop is not available in RedHat / CentOS)

with output like this

This example maybe is simple, but the point is that there is rich scripting language with tons
of probes you can intersect ( kernel functions, FS drivers functions, any other drives and modules)

What else I see very useful in SystemTap it can work in userspace. That is you can use it to profile your and any application that has -debuginfo packages ( all -debuginfo for standard RedHat RPMS you can download from RedHat FTP), but basically it is info you get compiling with gcc -g.

Well, there seems another war story going on. To profile userspace application with SystemTap your kernel should be patches with uprobes patch, which fortunately is included in RedHat based kernels, but not included in vanilla kernel yet. So I am not sure if you can get userspace profiling running in another distributives.

There is quite simple script that I tried to hack around MySQL ®

with output which I get running simple SELECT against InnoDB table:

Again, this case is maybe too simple, but basically you can intersect internal MySQL function and script (measure time, count of call, statistics) what you what. I did not figure out yet how to intersect C++ style function (i.e. ha_innobase::index_read), so there is area to investigate.

So I am going to play with it more and do some useful scripting to get profiling of MySQL.

And it seems SystemTap can re-use DTrace probes available in application, as you may know DTrace-probes were added into MySQL 5.4, so interesting how it works.

I should mention that there is second alternative of DTrace... It's .... DTrace port. Looking on blog it seems one-man project and currently author is fighting with resolving userspace issues. I gave to this a try, but on my current RedHat 5.4 after several runs I got "Kernel panic", so it's enough for now.

Entry posted by Vadim | 5 comments

Add to: | | | |

Here is result (in transactions per minute, more is better):

Here is graph of disk writes (column bo in vmstat) during benchmark

As you see noop / deadline can utilize disks much better.

For reference I used tpcc scripts from https://launchpad.net/perconatools, generated 100W (about 9.5GB of data on disk), and used next XtraDB params:

Entry posted by Vadim | 19 comments

Add to: | | | |

The problem could be repeated with SysBench as follows:

On Dell R900 with CentOS 5.2 and ext3 filesystem we get 1060 transactions/sec with single thread and sync_binlog=1 while with 2 threads it drops to 610 transactions/sec Because of synchronous serialized writes to binlog we're not expecting significant improvements in transaction rate (as the load is binlog fsync bound) but we surely do not expect it to drop ether.

To ensure this is not RAID controller related we repeated the run on the in memory block device and got very similar results - 2350 transactions/sec with 2 threads with sync-binlog=0 and just 750 transactions/sec with sync-binlog=1

We tried running data=writeback and data=journal just to make sure but results were basically the same.

Using XFS instead of EXT3 gives expected results - we get 2350 transactions/sec with sync_binlog=1 and 2550 transactions/sec with sync-binlog=0 which is about 10% overhead.

EXT2 filesystem also behaves similar to XFS so it seems to be EXT3 specific performance issue. We do not know if it is CentOS/RHEL specific and if it is fixed in the recent kernels. If somebody can run the same test with different kernels it would be interesting to know results.

You also may wonder how could this problem be unnoticed for a while ? First it only applies to the binary log. Innodb transactional log does not have the same problem on EXT3. I expect this is because Innodb log is pre-allocated so there is not need to modify meta data with each write while binary log is written as growing file. Another possible reason is small sizes of writes which are being fsync'ed with binary log. Though these are just my speculations - we did not investigate it in details.

Another reason is - it only happens with high transaction commit rate. If you're just running couple of hundreds of durable commits a second you may not notice this problem.

Third - sync-binlog=1 is an option great for safety but it is surely not the most commonly used one. The web applications which often have highest transaction rates typically do not have so strong durability requirements to run this option.

As the call for action - I would surely like someone to see if EXT3 can be fixed in this regard as it is by far the most common file system for Linux. It is also worth to investigate if something can be done on MySQL side - may be opening binlog with O_DSYNC flag if sync_binlog=1 instead of using fsync will help ? Or may be binlog pre-allocation would be good solution.

Entry posted by peter | 17 comments

Add to: | | | |

For any reason, you decided to run MySQL database on AIX 5L operating system. You compiled it successfully, configured and.. unluckily database didn't start due to memory allocation problem?

Basically, for some historical reasons AIX OS will allow your application to allocate maximum 256MB of memory per process by default. To use more, you have to use "Large Address Space" memory model so AIX will split memory into 256MB data segments, making possible to reserve more than one segment per single process.

You may set it up different ways:

• - using –bmaxdata=0xN000000 option for linker
• - setting MAXDATA value in LDR_CNTRL as run-time

[ LDR_CNTRL='MAXDATA=0xN000000' ] - where N is the number of segments.

So, to make your MySQL working fine:

export LDR_CNTRL=’MAXDATA=0×80000000′

It allows database to allocate 2GB of memory ( 8*256MB ) which is maximum value for 32-bit compiled applications.

But how to deal with 64-bit systems? It's quite easy, but you have to remember about setting OBJECT_MODE to 64 for both, self-compiled and binary distributed databases. After it, you will be able to set up bigger segments multiplier in MAXDATA variable at run-time.

export LDR_CNTRL='MAXDATA=0xFFFFFFFFFFFFFFFFFF'

Entry posted by Piotr Biel | 11 comments

Add to: | | | |

I also would like to hear what you do yourself, and why

The rationale behind disabling swap is what there is nothing you want to swap out on such box anyway and if you disable swap file kernel will not swap and possibly will be able to manage memory smarter knowing it does not need to look for pages to swap out or balance memory for reducing a cache or swapping something out.

And indeed if you run with swap disabled you would not have the issue of swapping happening on the box as there is just nowhere to swap.

So what is about enabled swap - if we assume the kernel is smart (as we did for second case) we will be able to save a bit of memory as there are some programs which are started but never never really used. These would be first valid candidates to swap out. Second if you have swap file you get a bit more flexibility. What happens if you miscounted something or there is a gotcha in your application and you ended up with a lots of connections creating large temporary tables and so MySQL running out of memory (and getting killed by kernel). It could not even be MySQL but some script cron script or something similar with same affect.

In practice there are additional issues for both of the configurations - if you keep swap enabled you may have hard time keeping MySQL in memory because Kernel would love to swap it out. Recent kernels become much better than few years ago but I still run in workloads which expose bad kernel behavior and swap buffer space out even with /proc/sys/vm/swappiness =0. You can lock MySQL in memory by using --memlock option but it opens its own can of worms.

If you keep swap disabled you can run into another problems. A lot of code in the kernel assumes there is a swap space and I've seen kernel starting to behave crazy when there is memory pressure and there is nothing to swap out. This may actually mean the buffer pool (or other caches) which you can safely use can be less if you have swap file disabled. Though I must note again this can be workload and kernel version specific.

Besides general considerations there are many case specific ones. For example if you would rather spend more memory (allowing more reserve for spikes) than to have slowdown caused by swapping activities in case memory overcommit occurs you may be better off without swap file. If on the contrary you want to use as much memory as possible to get last bit of performance and do not mind slowdowns if you mixedup as well as have tools to resolve them quickly (ie killing runaway queries) keeping swap file enabled may be right for you.

Myself I tend to keep swap enabled on MySQL server but of course ensure there is not swapping happening (si/so columns in VMSTAT are zero or close to zero).

If you want to keep swap file disabled that also fine but you should make sure your workload is stable so there are no large spikes in memory requirements, as well as you have processes and discipline to make sure it is kept stable.

Entry posted by peter | 18 comments

Add to: | | | |

When it comes to MySQL it is very useful with MyISAM tables which has their data file cached by OS cache only so you do not have any good information from MySQL side on what data is cached. You can also use it with Innodb to see how much memory are you wasting with double buffering by not using of O_DIRECT.

Besides general clues such as 50% of my file is cached you should watch for dynamics - for example check it during backup process and compare it due to normal load - this can give you a clue if slow down happens because of extra IO pressure or just because pages were washed out. You can also check how pages are cached. For example every second page cached may be helpful for point queries but does not save a lot of IO for doing table scans.

One thing I'm still missing is looking it from another side - so I have say 10GB of OS cache used on the server but how can I tell what is using it ? This look from another side would help me dramatically to find out what is causing cache pressure and what needs to be worked on. Scanning all files on filesystem and checking which are cached obviously does not work.

The fincore looks more like proof of concept tool - it is a bit simplistic, however being written in Perl it is easily hackable - if you want to make it to print percentage of file cached or "graph" showing how cached pages are distributed among file is very easy.

The great thing about this tool it is very fast and it does not disturbs OS file cache by using mincore function to get pages which are currently in cache.

This function is actually the real meat here - the tool is simplistic but it shows how to use the function so you can write real stuff. For example using this tool MySQL can easily add amount of cached data per table for MyISAM and Archive tables to INFORMATION_SCHEMA (or other system tables) which would be really cool Of course than one would need to implement cache content tracking for storage engines which cache everything in their own cache memory - Innodb, Falcon, Maria.

Having information about how large portion of table is cached would allow optimizer to take much smarter decisions in many cases.

Lets now see some examples:

So we can see one of Innodb log files is practically uncached while other has about 1/3rd cached - this makes sense, perhaps second log file is being written now and there is a "tail" of pages which just were not removed from the cache yet. As Innodb does not read logfile unless in recovery these are waste and Innodb could use fadvice to give instruction to kernel not to cache these as long as it can't perform direct IO to log files on Linux because it is not aligned.

Out of 250GB innodb data file none of pages are in cache - this is because this instance is using O_DIRECT flag to bypass data buffering and we can well see it works.

Lets now see stats for MyISAM tables:

The performance log for yesterday is almost out of cache. It is about 0:50 by server clock this is why we still can see some pages remaining. Today log file is 50% in cache. Knowing access pattern to the file you can draw some conclusions about how much IO pressure we have on this server.

P.S If you would hack this tool or know any similar tools please let me know.

Entry posted by peter | 9 comments

Add to: | | | |

The performance difference was huge - we could read single file at 80MB/sec while fragmented files only deliver about 2MB/sec - this is a massive difference.

The test was done on EXT3 and it looks like it does not do very good job preventing file fragmentation for large amount of growing files.

It would be interesting to see how other filesystems deal with this problem, for example XFS with delayed allocation may be doing better job.

I also would like to repeat the test with MySQL MyISAM tables and see how bad the difference would be for MySQL but I would expect something along those lines.

Interesting enough it should not be that hard to fix this issue - one could optionally preallocate MyISAM tables in some chunks (say 1MB) so its gets less fragmentation. Though it would be interesting to benchmark how much such approach would generally help.

Until we have this feature - reduced fragmentation is one more benefit we get with batching. For example instead of inserting rows one by one in large number of tables once can be buffered in memory (application or MyISAM memory table) and flushed to the actual tables in bulks.

Entry posted by peter | 10 comments

Add to: | | | |

The interesting thing this article points out is - there are multiple versions of each of the schedulers, while name remains the same. This means unless you really know mapping between kernel versions and scheduler versions it is very hard to evaluate benchmark results.

This could be noticed by benchmarks we've done over years. Long time ago "AS" scheduler could be several times slower than deadline for MySQL workloads such as SysBench or DBT2 when it went down to 30% difference and in the last runs we've done difference was not really significant.

This article also points out benchmarking IO schedulers you should look at more numbers than aggregate bandwidth - you also better to measure per client bandwidth as well as max latency as this is what can be the problem. Take a look at these old results for example. It also means you'd better to perform IO scheduler benchmarks on mixed load with different of task, for example mixing OLTP with some reporting queries if you really want to see the difference.

From the article it looks like CFQ should be good choice for databases and it is also found to work pretty well by some benchmarks we've done. The only question if it is doing as good as it could - In the docs it is mentioned it uses "per process" scheduling while MySQL is single process but single thread - does each thread gets its own queue in reality or is it shared ?

We should look into this when we'll run more benchmarks for IO Schedulers.

Entry posted by peter | 6 comments

Add to: | | | |

The crash with many threads in FreeBSD is known problem and is not MySQL fault. More info is available in FreeBSD bug report

I'm not big expert in FreeBSD and did not saw http://wikitest.freebsd.org/MySQL before. This page recommends to use libthr instead of libthreads.
The results with libthr looks better:

Interesting thing with 4-64 threads FreeBSD is better than Suse in InnoDB benchmark. I think it is related to InnoDB's implementation of syncronious primitives. For MyISAM Suse is stable better.

Configuration params:
Box: Dual Core Athlon 3800+, 1Gb of RAM, Motherboard ASUS A8N-E

MySQL 5.0.22

params for InnoDB:
--innodb-buffer-pool-size=500M --max-connections=500

params for MyISAM:
--key-buffer-size=500M --max-connections=500 --skip-innodb

Suse 10.0:
kernel-smp-2.6.13-15.x86_64
NTPL

Schedulers comparsion.
By request I made tests with 4BSD scheduler:

Interesting with 256 threads BSD scheduler looks better.

Entry posted by Vadim | 14 comments

Add to: | | | |

Some results for MyISAM table, query SELECT id FROM sbtest WHERE id BETWEEN N AND N+20000, id - primary key.
Boxes: Sun V40z, Solaris 10, 4 x Dual Core Opteron @ 2.2GHz (8 logical cpu), 16GB of RAM, StorEdge 3310
and Quadxeon, RedHat AS 3, 2.4.21-15.Elsmp, 4 x Intel(R) XEON(TM) MP CPU 2.00GHz, 4GB of RAM, SATA RAID 10
The results in queries per sec (more is better)

So if you are using range-queries and doing INSERT not often than disabling concurrent-insert with --skip-concurrent-insert can improve MyISAM performance.

As I said the reason of bad scalability is rw-locks and currently MySQL developers are working on CPU-depended rw-locks implementstation, this will be available in MySQL 5.1 - 5.2

Entry posted by Vadim | 9 comments

Add to: | | | |