MySQL Performance Blog

This is mainly a cheat sheet for me to remember. Nothing rocket science.

It often makes sense to use netcat/nc to copy MySQL database between hosts in trusted networks. It bypasses encryption overhead of SSH and depending on configuration can be significantly faster.

Also note MySQL should be down when you copy data unless you’re copying from snapshot etc.

So to copy go to the mysql data directory on both boxes; such as cd /var/lib/mysql . Make sure target directory is empty. Now on the TARGET server do nc -l 4000 | tar xvf - and on the SOURCE server do tar -cf – . | nc target_ip 4000

Also note – the port you’re using should be open in the firewall.

I find it very interesting how Sun does not get the very basic principle of true community Open Source development – you’ve got to give up on making a big splash.

Traditional close source company often develop product in the secret and when it comes out as a surprise for computers and making a big splash for the users. Does it remind you something ? Yes! this is exactly how Innodb Plugin was released last year or MySQL 5.4 performance improvements this year.

Community did not know about them and did not participate early in this efforts.

Another big splash which seems to be planned later this year is “Performance Schema” – which is in development for years as this post claims but to date there is no code for community to play with
[read more...]

Every so often I run into situation when I need to kill a lot of connections on MySQL server – for example hundreds of instances of some bad query is running making server unusable. Many people have special scripts which can take the user, source host or query as a parameter and perform the action. There is also a way to do it just using MySQL with a few commands:
[read more...]

We announce next version of our xtrabackup tool and we consider it stable enough to put label RC on it.

Changelist includes:

- use O_DIRECT by default for handling InnoDB files
- use posix fadvise call to disable OS caching of copying files
- disable recovery of double buffer

Also we added binary builds for FreeBSD 7 64bit platform

You can download current binaries (64bit) RPM for RHEL4 and RHEL5 (compatible with CentOS also), DEB for Debian/Ubuntu, tar.gz for Mac OS / Intel and .tar.gz for FreeBSD 7:
http://www.percona.com/mysql/xtrabackup/0.7/.
By the same link you can find general .tar.gz with binaries which can be run on any modern Linux distribution.
By the same link you can download source code if you do not want to deal with bazaar and Launchpad.

The project lives on Launchpad : https://launchpad.net/percona-xtrabackup and you can report bug to Launchpad bug system:
https://launchpad.net/percona-xtrabackup/+filebug. The documentation is available on our Wiki

For general questions use our Pecona-discussions group, and for development question Percona-dev group.

For support, commercial and sponsorship inquiries contact Percona

Using xtrabackup for copying files can really saturate your disks, and that why we made special option --throttle=rate to limit rate of IO per second. But it really works when you do local copy.
What about stream backup ? Even you copy just to remote box with
innobackupex --stream=tar | ssh remotebox "tar xfi -"
, read may be so intensive so your mysqld feels impact, slave getting behind, etc…

For this there is a nice small utility – pv .

With pv you run:
innobackupex --stream=tar | pv -q -L10m | ssh remotebox "tar xfi -"

and it will limit channel rate to 10 M per second.

I should say I can see results of new engineering/refactoring/bug hunt efforts inside Sun/MySQL.
Over last couple of weeks I started getting a lot of messages from the bugs system about bugs I reported long ago which were deferred to be fixed later or were left in open state. Here is example of such a bug.

I really hope this effort will result in a lot of these old annoyances fixed, which really matches my vision for MySQL – we do not need more big features we need old ones to work well and be convenient.

Sometimes when you do profiling you can see number like this in timestamps. Periodically in our patches there was Query_time: 18446744073709550.000 in slow.log file (well, it was fixed recently, but still appears in other places).

I faced this problem several years ago when only 2-core AMD Opteron systems appeared and I noticed sysbench getting crazy showing query execution time 18446744073709550.000 or like this.

Obviously this is unsigned integer which was received by subtraction bigger number from smaller.
But how it can be if we use:

start_time = clock_gettime(CLOCK_THREAD_CPUTIME_ID, &tp);
…
query_execution
…
end_time = clock_gettime(CLOCK_THREAD_CPUTIME_ID, &tp);

total_time = end_time – start_time;

How we can get query executed in negative time ? It would be too good to be real .

After some research I found the reason, and it is interesting that I want to share it even few years later. The reason is that different CPU on the same motherboard uses different frequency generators (each CPU has its own frequency generator, though exact implementation depends on CPU vendor), and there is no way to get these generators synchronized. That is if in your server you have 4 CPU, each of them lives on its own time. Usually difference between generators is very small (microseconds matter), but it appears this difference is enough to get negative time.
It happens when your query started on one CPU and during execution it was switched to another CPU (where generator delayed to first CPU) and time between two measurements were small enough to catch difference in generators. In result you have negative time, but it actually, it is more serious, it mean – we can’t get exact time between two points in application.
So designing profiling in your application be aware of. I wonder how realtime sensor systems, where microseconds are important, deal with this.

I want to make this Friday a bit more interesting – how do you feel to train a bit in InnoDB data recovery techniques.
I have .ibd datafile which was created by query
CREATE TABLE `tryme` ( `email` mediumblob, `content` mediumblob ) ENGINE=InnoDB … (SOME PARAMETERS SKIPPED) …;

and I inserted one record into this table with email address and English sentence into field (content). Some transformation where applied to both fields so just HEX viewer may not help you there.
Size of meaningful de-transformed content is
length( … (content) … ) = 15.

Can you recovery both email and content fields ? To add some competition I want to give a prize for winner, first who sends content (meaningful English sentence) to given email – will receive our book “High Performance MySQL” 2nd edition, or if he has already this book or just hate it by some reason – I can propose Amazon 25$ Gift Card as replacement.

File tryme.ibd you can download here http://mysqlperformanceblog.com/files/contest/tryme.ibd. Size 65536 bytes. md5sum a53d3ee3f3a33854f21b5f550ae53e18.

I see a lot of people filtering replication with binlog-do-db, binlog-ignore-db, replicate-do-db, and replicate-ignore-db. Although there are uses for these, they are dangerous and in my opinion, they are overused. For many cases, there’s a safer alternative.

[read more...]

I do not know if you noticed it, but Google (Mark Callaghan, Justin Tolmer and their internal mysql-team) made a great contribution to MySQL. Patches global transaction IDs, binlog event checksums and crash-safe replication state are separated and published on Launchpad (https://code.launchpad.net/~jtolmer/mysql-server/global-trx-ids).

For me it was a big wall in using these patches that they were part of one big patch, which you can apply only to 5.0.37, and now there is no barrier to include patches into our builds or MySQL releases.

If you do not know what is Global Transactional ID is – it is worth to look http://code.google.com/p/google-mysql-tools/wiki/GlobalTransactionIds. From my point of view – it is absolutely new view on MySQL replication and it can change MySQL replication architecture.

We definitely will look if we can integrate patches into percona builds and provide binaries if there is no problems. Also I mostly sure the patches will be included into MariaDB along with other Percona-improvements and XtraDB storage engine.