When it finished on destination server you run

And you have ready database directory, just copy my.cnf from original server and start mysqld.

Entry posted by Vadim | 3 comments

Add to: | | | |

My research gave me next list: pigz (parallel gzip), pbzip2 (parallel bzip2), qpress ( command line utility for QuickLZ) and I wanted to try LZO (as lzop 1.03 command line + LZO 2 libraries). Actually lzop does not support parallel operations, but it is know to have good decompression speed even with 1 thread. UPDATE 17-Mar-2009: I added lzma results also by request from comments.

For compression test I took ~12GB of InnoDB data files generated by tpcc benchmark with 100 warehouses.

I tested 1, 2, 4 parallel threads for tools that support it and different level of compression ( 1,2,3 for qpress; -1 and -5 for other tools)

The raw results are available here http://spreadsheets.google.com/ccc?key=pOIo5aX59b6biPZ0QTVMXHg&hl=en, and I copy table in place in case if Google stops to work.

To summarize results:

• pbzip2 obviously show good compression, but the speed of processing is too slow. What is interesting on Level 5 the compression is worse than in pigz Level 5
• pigz is good for compression and faster than pbzip2 but still not so fast; however multi-threaded processing may be OK, especially if you need to keep compatibility, e.g. copy result on boxes where only standard gzip available
• qpress is not so good in compression ration, but speed is impressive, and maybe we will ship xtrabackup with this compression
• LZO is even faster in decompression than qpress, but I would like to see parallel version. There is the patch for it, but it did not apply clean to lzop 1.02, so I skipped it
• In my opinion in all cases Level 1 of compression shows better tradeoff between size of archive and compression/decompression time

There is no obvious winner, it depends on what is more important for you - size or time, but having this data we can make decision.

Entry posted by Vadim | 18 comments

Add to: | | | |

My question in such case is always what exactly are you trying to get by using SAN ?

Depending on the goals the advice could be a lot different.

Manageability SAN allows you to manage storage easily, compared to directly attach storage. If you have one box which starts to require more IO you can have more resources allocated to it. This is pattern which is somewhat typical for Enterprise managing a lot of relatively small MySQL installations or managed hosting provider. For large MySQL web MySQL installations using Sharding or replication IO needs are typically rather predictable. There are also often nice monitoring tools available to see disk utilization latencies queues etc. There is however also a downside compared to directly attached storage in terms of performance management - the SAN is shared resource (has more or less shared parts) so it is possible one application to impact other which means you can't always analyze the local system performance independently from anything else.

Performance This is perhaps the most interesting one. I constantly see SANs sold as magic equipment which magically going to solve all performance problems, with magically means nobody knows how exactly. I would suggest always to question yourself where do you expect these performance gains to come from ? If we're speaking about purely disk based SAN (no Flash) the drives are same drives you will see in directly attached storage and each can do only so many IO requests per second. True the SAN can have much more hard drives than directly attached storage but usually also at much higher prices per drive. The next possible advantage is software - can SAN have some very smart software instead of simple RAID you use with directly attach storage which can magically improve performance ? There are cases when it can but you surely can ask this question. For example if you share same physical drives among applications which have different peak usage patterns or something similar there can be an advantage. The question is again if it is high enough to pay for the price premium ? The third one is caching. SAN can have a lot of cache though servers can typically have more. If you can afford SAN you should afford 128GB of memory or so on the server too which will make read caching more efficient in its memory while write buffering can too be done by MySQL and local RAID (with BBU) quite efficiently. It is worth to say there is a benefit of read caching on SAN - if you have MySQL/server restart you may have warmup being quite shorter than in case of local storage.

It is also worth to note SAN does not only have advantages compared to directly attached storage but also downsides - SAN typically has better throughput (because of larger amount of drives) but longer latency because extra processing (and extra trip) involved. This in particular hurts log writes which are very latency critical.

Now what is about MySQL/Innodb specifics ? First, in MySQL if you're looking for durable transactions the log write latency is triple important compared to other database systems. MySQL has to do more than one fsync() for the log because it internally implements XA to synchronize with binary log and also if you're looking for maximal data consistency (as SAN users often are) you better have binary log flushed on commit too. MySQL also has broken group commit (which we have partial fix for) meaning concurrent transaction commits will need to be serialized.

Second, MySQL is often not able to submit a lot of outstanding IO requests which is needed to utilize SAN efficiently. It is especially problem if you're running replication Slave as slave thread will issue single IO request most of the time making it sensitive to latency rather than throughput. Also you should plan on any queries executed having only one IO at the time if Innodb or MyISAM tables are used. There is read-ahead functionality but usually it is not able to drive number of outstanding request significantly. For write intensive workload you will have problem with number of outstanding writes too. Though for that we have a fix at least.

Scalability We need more IO or space than we can get from 6-8 build in drives... so lets do SAN is the story here. As I mentioned in Performance section you may be surprised performance is not "scaling" as much as you expected. You should also consider external directly attached storage which is cheaper alternative in most cases and is especially good if your IO needs are predictable, like you need 3TB of space per box - just external disk enclosure does it great. My Advice for Scaling IO capacity with SAN (this applies to directly attached storage too) is to really understand what you're trying to scale and analyze things appropriately. I've seen in so many times this was a poor choice as it did not allow to scale (ie attempt to scale replication) or was a poor choice, with adding memory or SSD being a better choice.

High Availability Some people are got used to using SAN based active-passive clusters for availability purposes and they look to do the same with MySQL. This works though I do not think it is the best choice. SAN is just another object which can fail completely. Sure it is more reliable, same as server with multiple power supplies ECC memory kill-chip and RAID is more reliable than without these technologies but I always in my HA architectures it is a single "point" which can experience "failure". Note even if hardware fully redundant the Firmware may have a glitch causing failure and data corruption (and this is not just a speculation, some clients really had it)

Even if you do not run any "Cluster" having SAN you can always "connect" the storage to another node - if server fails or if you just want to upgrade to bigger box, which can be more complicated with directly attached storage.

If you want SAN like high availability I would highly consider DRBD which can do storage level replication between directly attached volumes. Now with support for Dolphin Interconnect it can also offer very decent latency. It also have a benefit of having the system which you can actually split in two (ie for recovery purposes). Sure it needs double the storage but you get 2 copies of data too and using SAN you probably pay more than 2x premium anyway.

Yet another approach is to use MySQL replication with something as MMM or Flipper. I think this actually works best for most cases (unless async nature of MySQL replication is a showstopper) and it also solves the other big pain of semi-online MySQL/OS upgrades and Schema changes.

Backups SAN has number of advantages for operations (and I guess these are the guys which often push for it). It often would offer snapshots with low overhead allowing to take a backups conveniently. Also if you keep last uncompressed backup on SAN (or last snapshot) you may be able to minimize recovery time significantly by switching to that backup - no data copy will be required which can be significant concern to 1TB+ databases.

My Take: I view SAN as solution for niche circumstances when it comes to MySQL. It may be you need it but it is not silver bullet and best solution for all problems at once. When evaluating SAN you should also evaluate external directly attached storage, SSDs, Increasing system memory as well as sharding and replication as a scaling solutions.

Entry posted by peter | 18 comments

Add to: | | | |

In short words using Intel SSD X25-E card with enabled write-cache (which is default and most performance mode) does not warranty storing all InnoDB transactions on permanent storage.
I am having some déjà vu here, as Peter was rolling this 5 years ago http://lkml.org/lkml/2004/3/17/188 regarding regular IDE disks, and I did not expect this question poping up again.

Long story is:
I started with puting XFS on SSD and running very primitive test with INSERT INTO fs VALUES(0) into auto-increment field into InnoDB table. InnoDB parameters are

Actually most interesting one are innodb_flush_log_at_trx_commit=1 and innodb_flush_method = O_DIRECT (I tried also default innodb_flush_method, with the same result), using innodb_flush_log_at_trx_commit=1 I expect to have all committed transactions even in case of system failure.

Running this test with default XFS setting I saw SSD was doing 50 writes / s, this is something so forced me to check results several times - come on, it's SSD, we should have much more IO there. Investigations put me into barries/nobarriers parameters and with mounting -o nobarrier I got 5300 writes / s. Nice difference, and this is something we want from SSD.

Now to test durability I do plug off power from SSD card and check how many transactions are really stored - and there is second bumper - I do not see several last N commited transactions.

So now time to turn off write-cache on SSD - all transactions are in place now, but write speed is only 1200 writes / s, which is comparable with RAID 10

So in conclusion to warranty Durability with SSD we have to disable write-cache which can affect performance results significantly (I have no results on hands, but it is to be tested).

What about LVM there ? Well, we often recommend to use LVM for backup purposes (even recent results are bad, we have no good replacement yet) and I tried LVM under XFS. With write-cache ON and default mount options (i.e. with barrier) I have 5250 writes / s, this is because LVM ignores write barriers (see http://dammit.lt/2008/11/03/xfs-write-barriers/ ), but again with enable write-cache you may lose transactions.

So in final conclusion:
1. Intel SSD X25E is NOT reliable in default mode
2. To have durability we need to disable write cache ( with following performance penalty, how much we need to test yet)
3. Possible solution could be put SSD into RAID controller with battery-backup-ed write cache, but I am not sure what is good ones - another are for research
4. XFS without LVM is putting barrier option which decreases write performance a lot

Entry posted by Vadim | 26 comments

Add to: | | | |

Performance - The performance comes down to performing up to users expectations (or expectations of your boss) which drills down to two important metrics - response time also sometimes called latency and throughput also called capacity. Response time applies to even single user and mostly critical in applications when a complex task needs to be performed such as report generation or search. It is just unacceptable if search takes a minute even if system has a capacity running for 100 of them at the same time. The response time is the most important aspect of performance because when we define system capacity we also define conditions on response time. For example system can handle 100 transactions a second with 95 percentile response time of 1 second.

Scalability - In the high volume applications we assume scaling to larger amount of users, larger systems, larger database size and this is what I will be speaking about. There is also aspects of scaling down for embeddable systems but it is whole other story. The question of scalability usually comes because system needs to handle larger load (or operate with larger database) which comes to the question of performance on the same system, getting larger box "scaling up" or using multiple systems "scaling out". Scalability is really funny term because it is used in some many different context. You may hear "It does not scale" applied to the application on same system as load or database size increases as well as to the no significant response to getting more hardware. So I would define Scalability as ability to increase or sustain performance as response to the environment changes.

Efficiency This is my favorite one. Your application may be performing well and scaling to insane load and data size but If I can do the same with half the hardware (or half development costs) my application is more efficient than yours. The complexity of Efficiency term comes from the fact you can set the optimization goals very differently. Some are concerned about development time and operations complexity, others about hardware and software cost and TCO and third about power consumption.

What is interesting in many cases I see the focus on Performance and Scalability while not really focusing on efficiency a lot (at least until VC money run out). I'm constantly being surprised how efficient and inefficient systems you can find among top web sites. It is not an exception to see the difference in efficiency as much as 10 times or sometimes even more.

When you evaluate different solutions, see some suggestions in the internet I'd urge you to use all 3 metrics above for evaluation (using efficiency metric which is most critical for you right now). It is quite possible the most efficient solution somebody have picked is efficient using different metric compared to what you need.

Entry posted by peter | No comment

Add to: | | | |

There are 2 things you should ask yourself. First is the scale comparable - the recipes from Facebook, YouTube, Yahoo, are not good for like 99.9% of the applications because they are not even remotely close in size and so capacity requirements. Second if this "smart thing" was truly thought out architecture choice in beginning or it was the choice within code base constrains they had, and so you might not have.

Let me look into couple of most typical reservations.

Sharding - This is perhaps the technology people get obsessed with most regularly. Sometimes it looks like a homepage running on 100K database visited by 100 people in a month is attempted to be sharded. Remember as commodity hardware is advancing the size of the application when you really need to shard moves further and further away. I remember LiveJournal with 4GB of memory per box doing sharding 5 years ago.... well now you can get a box with 128G of RAM within $15K. Keeping working set in memory is not the only reason for sharding but one of the most frequent ones. The examples I like to use is YouTube - they did not shard until after Google bought them (though they were in pain) and 37Signals

When doing Performance Audit we tend to look at the required capacity and data size within current horizon. In many cases even with super optimistic assumptions application will do just fine with single "cluster" even on the current hardware for several years.

Replication Optimization People often get scared with the fact replication is single thread and often becomes bottleneck so they are using various optimizations including tricky prefetch approaches suggested by YouTube. Interesting enough this often happens even when system is far from reaching its replication capacity.

I would suggest measure and monitor your replication capacity (how long will it take a slave to catch up 1 hour lag of peak traffic ?) and act appropriately. Also focus on simple optimizations first, if you need to get to prefetch you're quite likely beyond reasonable use of single master and should have done sharding functional partitioning or something else.

Complex Replication There are impressive numbers out there on how many slaves people run and how complex replication topologies with multiple tiers filtering and writes to some intermediary slaves people use. For me simple is best. Complex architectures are more error prone harder to maintain (upgrades etc) and troubleshoot. Remember for every single "role" in such setup you need to understand what to do with it if any other "role" in the system fails, which escalates complexity. You may need something more advanced than master and one slave but any complication needs to be justified. I also should note slaves are not overly efficient beasts - they not only store the copy of data on the disk, wasting resources but their caches are also highly redundant defeating the fact you may have a lot of total memory on the slave farm.

Reading from the Slaves The story heard is typically - Web applications often have significantly prevailing reads so to scale we better have many slaves which we can use to handle most of our read traffic right ? Sure. Unless you're using memcache or other caching option. Successful memcache implementations often report 90% cache hit ratio meaning 10 to 1 read ratio drop backs to one to one. This means you may not need a lot of slaves if your application allows use of efficient caching.

Now lets look at the simplified case - you got pair of servers replicating as Master-Master which you typically want for high availability and online schema changes. How far you want to go making your application being capable to read from the slave ? Remember as you're doing this for high availability and online schema changes you're planning to operate without slave every so often, meaning one server should be able to handle all traffic from capacity planning standpoint anyway. At the same time slave can be perfect to be used for non production impacting things like analytics.

High Availability The trick with high availability is the more complex architectures and processes you use for high availability prevention the more likely it is for them to fail. Unless you're Google scale with failures happening daily you can't really be sure you're handling "wild" failures, not the test ones well. Furthermore you always have to look at failures caused by other things - wrong code pushed to production, hacker break in, data center power failure etc.

Google guys tells us single MySQL server on a good hardware has MTBF somewhere between 1000 and 2000 days. This is a lot of time which means for most of applications having a pair of slaves (even though second slave is available for failover only 99% of the time) is more than enough.

I would say more. In my experience the availability of the application is only related to the MySQL redundancy for very high quality/high scale applications. I've seen applications having no downtime running for years on single MySQL server (which just does not crash) as well as complex no single point of failure database backend with application constantly going down because of bad code or something unpredicted.

Summary: So am I denying all MySQL industry practices (which we also covered in a great depth in our book) ? Not really. I'm just suggesting do not just grab advice from the Internet or friends tip and do not complicate beyond the need. You may start with couple of replicated nodes for high availability and maintenance if you're in serious business (and just one server and good point in time backup if you're on the budget) and assess any need for any complications. It may be boring but boring systems often have highest uptime

Entry posted by peter | 7 comments

Add to: | | | |

Innodb Hot Backup is a good working solution but it is not Open Source and so we can't ensure it will support all XtraDB features - extra undo slots, rollback segments etc.

After considering all alternatives we decided to develop Percona Xtrabackup tool, which will use same backup approach as InnoDB Hot Backup, that is taking online copy of InnoDB table files and transactional logs. This is going to be fully OpenSource, GPL licensed backup solution for MySQL (MyISAM and InnoDB for now).

We gathered basic ideas here http://www.percona.com/docs/wiki/percona-xtrabackup:start, and for the first stage it will be just full backup of files. For next stages we are looking to implement advanced functionality like incremental backup (copy only changes from last backup),table-level backup (copy only specified tables), streaming and parallel copying. Source code will be posted to Xtrabackup Launchpad project (https://launchpad.net/percona-xtrabackup/)

We call for more ideas you would like to see implemented (please leave them in comments). We got sponsorship for the base implementation work, but we're still looking for sponsorship for additional features, contact us http://www.percona.com/contacts.html if you are interested. Percona will also significantly invest in this project.

Entry posted by Vadim | 8 comments

Add to: | | | |

For table "article" itself you would notice the issue quickly if you would forget to alter it as inserts will fail because of primary key collisions. But for linked tables like above inserts work you just get a warning:

And in my experience very few people mind to check the warnings promptly.

So two things happen, one is nastier than the other. First the comments end up associated to the wrong article (4294967295) which can be quite a problem in particular if these are private comments. The other thing which tends to happen - you get A LOT of comments associated with this article and few systems are designed with YouTube capacity to handle millions of comments per item.

Note the question in this case is not like there is just one article which has insane number of comments but a as a all articles outside of allowed range have it:

MySQL optimizer behaves rather strange in such case (at least in MySQL 5.0) - instead of instantly saying there is no such value (because value is out of range what could be stored in the column) it truncates the value, performs index lookup, scanning all rows (just 3 in my trivial example) and when filtering them out because value really does not match the where clause.

This makes such error even harder to catch - you will not see wrong comments for a lot articles, you will see 0 while queries will just run insanely slow.

So how to avoid this problem ?

It is actually quite easy - you just need to enable strict mode:

Note however strict mode should not be taken lightly - it is good for reliability but many applications will need to be fixed before they start working properly with this mode.

Entry posted by peter | 4 comments

Add to: | | | |

I generally prefer to see Innodb as the main storage engine because it makes life much simpler in the end for most users - you do not get to deal with recovering tables on the crash or partially executed statements. Table locks is no more problem, hot backups are easy, though there are some important things which we have to consider on case by case basics before recommending the move.

Is MyISAM used as default or as a choice ? This is the most important question to ask upfront. Sometimes MyISAM is there just because it is default, in other cases this is deliberate choice with system being optimized to deal with MyISAM limits, for example there is a dedicated slave available for all long reporting queries. In case MyISAM was chosen not just happened to be it is important to build the good argument to suggest Innodb.

Application Readiness Application should be ready to work with Innodb, for example be ready to deal with deadlocks which can happen with Innodb even if you do not use transactions, but which are not existent with MyISAM. QA has to be performed as part of the move.

Performance Innodb has a lot to offer in terms of performance - Performance benefits and drawbacks. On the benefits side we usually see clustering by primary key, caching data, higher concurrency, background flushes while on the drawbacks side we see significantly large table size (especially if data size is close to memory size), generally slower writes, slower blob handling, concurrency issues, problems dealing with very large number of tables, slow data load and ALTER TABLE and others. Another big one is COUNT(*) without where clause which is often the show stopper for them move until it is worked around.

Operations What is good for MyISAM kills Innodb, such as copying binary tables between the servers. It is important the team understands Innodb and knows how to handle it, or be able to learn it. It is also important to adjust processes as required to work with Innodb. For example binary copy of one of the databases from the Slave to the dev envinronment works great for MyISAM but does not work with Innodb. Backup tools like "mysqlhotcopy" does not work etc. Note Performance also affects Operations aspects a lot - for example using mysqldump as a backup may well work for MyISAM but will start taking way too much time to do restore for Innodb. On large scale installations mysqldump does not work anyway but it may still work for you when you're running MyISAM but instantly break upon upgrading to Innodb.

Features The MyISAM features which forbid moving to Innodb are typically Full Text Search and RTREE indexes/GIS with Full Text being much more common. There are workarounds for both of them, including dedicated MyISAM slave or shadow table but it is important to consider them.

How about Mixing Storage Engines ? Sure you can mix storage engines but I suggest you doing is wisely. It complicates operations tasks (backups, balancing, performance analyzes) as well as it exercises not so common paths in the MySQL server - in particular Optimizer may have harder time because costs between storage engines may not be well balanced or replication of mixed table types which is quite complicated.

I prefer to pick one storage engine (typically Innodb) and when use other tables when it really gives substantial gains. I would not switch table to MyISAM because it gives 5% performance improvement but I can perfectly use MyISAM (or Archive) for logging.

Innodb Needs Tuning As a final note about MyISAM to Innodb migration I should mention about Innodb tuning. Innodb needs tuning. Really. MyISAM for many applications can work well with defaults. I've seen hundreds of GB databases ran with MyISAM with default settings and it worked reasonably. Innodb needs resources and it will not work well with defaults a lot. Tuning MyISAM from defaults rarely gives more than 2-3 times gain while it can be as much as 10-50 times for Innodb tables in particular for write intensive workloads. Check here for details.

Note: As Few people questioned me, I indeed forgot to clarify the scope here - I'm mainly speaking about OLTP/ Traditional web applications. for Analytics things are a lot different.

Entry posted by peter | 34 comments

Add to: | | | |

Today we announce release 1.0.2-2 of our XtraDB storage engine.

Here is a list of enhancements:

• split-buffer_pool_mutex

The patch splits global InnoDB buffer_pool mutex into several and eliminates waitings on flush IO and mutex when there is no enough free buffers. It helps if you have performance drops when data does not fit in memory.

• innodb_extra_undoslots

InnoDB has a concurrent transaction limit of 1024 because in the standard InnoDB the number of undo slots is fixed value. This patch expands the maximum number of undo slots to 4072 and allows better utilizing modern hardware. (Thank SmugMug for this feature!)

ATTENTION: If the option was enabled and the expanded slots are used, you cannot use the datafile with system tablespace for the software which assume 1024 slots (e.g. the other mysqld or innodb hotbackup). See documentation for details.

Percona XtraDB 1.0.2-2 available :

• in source code from Launchpad, the version 1.0.2-2 you can get as bzr branch lp:percona-xtradb -r tag:release-1.0.2-2 percona-xtradb-1.0.2-2
• as source code in tar.gz
• as binaries, percona xtradb is compiled in into MySQL-5.1.30 RPM
• as separated shared library, to use as drop-in plugin for existing MySQL-5.1.30 installation

XtraDB is compatible with existing InnoDB tables (Warning: unless you used innodb_extra_undoslots ) and we are going to keep compatibility in further releases. We are open for features requests for new engine and ready to accept community patches. You can monitor Percona's current tasks and further plans on the Percona XtraDB Launchpad project. You can also request features and report bugs there. Also we have setup two maillists for General discussions and for Development related questions.

Entry posted by Evgeniy Stepchenko | 9 comments

Add to: | | | |