Unlike MyISAM Innodb does not have to keep open file descriptor when table is open – open table is purely logical state and appropriate .ibd file may be open or closed. Furthermore besides MySQL table_cache Innodb maintains its own (called data dictionary) which keeps all tables ever accessed since table start – there is no variable to control its size and it can take significant amount of memory in some edge cases. Percona patches though provide innodb_dict_size_limit to restrict growth of data dictionary.

So I started with same series of test and creating 100.000 tables with single integer column. The process of creating tables took about 45 minutes which is a lot more than MyISAM and the total size on disk was 12GB in .ibd files plus some space allocated in system tablespace. So if you create Innodb tables you better store some data in them otherwise there will be a huge waste of space.

I used MySQL 5.4.2 for tests which should be as good as it gets in terms of optimizations in this space.

To keep test alligned to my previous experiments I was running with table_open_cache=64 and tried innodb_open_files=64 and 16384.

Reading 100.000 tables first time after MySQL time takes about 500 seconds (so it is some 200 tables/sec) – first time Innodb actually populates data dictionary. The second time we do same operation it takes about 25 seconds (4.000 tables/sec) which is quite a difference. As we can see even in case table it fully in Innodb data dictionary the operation is slower than MyISAM tables. Though the difference can be related to the size of set of empty tables which is about 10 times smaller for MyISAM.

I found no significant difference whatever limit of open files was, which is not surprising as logical operation of opening file is rather fast on local file system – one can open/close file hundreds of thousands times per second.

To verify this I tried doing “open table” test for only 10K out of 100K tables – the performance was about the same, taking 1sec (on the second time) . Whenever innodb_open_files_limit was 64 (virtually all misses ) or 16384 (all hits) performance was the same.

As I mentioned Data Dictionary can take considerable amount of memory – In my case after reading all tables I got “Dictionary memory allocated 392029720″ which means very simple single tables takes about 4KB of space in data dictionary. More complicated tables can take a bit more.

So innodb_open_files does not affect performance a lot on reads – what is about writes ? I tried again very simple test inserting the row in each of 100K tables. This test ran about 180sec first time and about 260sec second time (with innodb_flush_log_at_trx_commit=0) go giving 550 and 380 updates/sec appropriately. Why was second time slower and not faster ? Because on the second run there were a lot of dirty pages in innodb buffer pool which had to be flushed before recycling. First ran however was done with clean buffer pool (after reading all tables once)

Same as with select case I could not see any measurable difference between two tested innodb_open_files values.

Finally I decided to test the crash recovery – does it make any difference ?

The crash recovery in Innodb is nasty if you have a lot of tables:

091118 18:43:36 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files…
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer…
InnoDB: Doing recovery: scanned up to log sequence number 12682768136
091118 18:47:44 InnoDB: Starting an apply batch of log records to the database…

If Innodb detects it is not shut down properly it will scan all .ibd files which took a bit over 4 minutes for 100K tables but which obviously can take a lot more if there are more tables or they are less cached than in this case. This part of data recovery does not depends on amount of records which need to be applied just about number of tables.

With innodb_open_files=64 I got bunch of warning messages during recovery:

091118 18:47:44 InnoDB: Warning: too many (67) files stay open while the maximum
InnoDB: allowed value would be 64.
InnoDB: You may need to raise the value of innodb_max_files_open in
InnoDB: my.cnf.
InnoDB: fil_sys open file LRU len 0
091118 18:47:44 InnoDB: Warning: too many (68) files stay open while the maximum
InnoDB: allowed value would be 64.
InnoDB: You may need to raise the value of innodb_max_files_open in
InnoDB: my.cnf

So we can see Innodb may with to have so many open files during recovery stage and it will open more files than allowed if needed.

Both scanning open files and applying logs took about 9 minutes in this setup. This number of course can change a lot depending on hardware log file size workload and even when crash happen (how many unflushed changes we had)

Repeating test with innodb_open_files=16384 I got about same crash recovery speed though with no warnings.

So it looks like innodb_open_files_limit=300 is not being that large liability even with large number of tables and you can also safely increase this number if you like – there is no any surprises such as surprised slow downs for replacing open files in the list. I guess Heikki knows how to implement LRU in the end

Entry posted by peter | 2 comments

Add to: | | | |

So let me introduce problem at first. The table is

Table has 11864696 rows and takes Data_length: 698,351,616 bytes on disk

The problem is that after restoring table from mysqldump, the query that scans data by primary key was slow. How slow ? Let me show.

The query in question is (Q1):

SELECT count(distinct username) FROM tracker where TIME_ID >= '2009-07-20 00:00:00' AND TIME_ID <= '2009-10-21 00:00:00' AND (tracker_id=437)


On cold buffer_pool, it took:

However the query (again on cold buffer_pool) (Q2)

SELECT count(distinct username) FROM tracker where TIME_ID >= '2009-07-20 00:00:00' AND TIME_ID <= '2009-10-21 00:00:00'


Difference is impressive. 4 min 13.61 sec vs 18.81 sec

If you want EXPLAIN plain, here it is:

For Q1:

For Q2:

Query Q1 is executed using Primary Key, and Query Q2 is using block_id key.

To get more details I ran both queries with our extended stats in slow.log (available in 5.0-percona releases)

So for query Q1:

And for query Q2:

As you see for Q1 IO read took 236.149003 sec vs 0.165124 for Q2. But Q1 is scan by primary key, which supposed to be
sequential!

Let's see on another statistic, which available in innodb_check_fragmentation patch:

for Q1:

and for Q2:

So you see for Q1 it was not sequential scan, even it is primary key, but it is sequential for Q2.

So what's the answer ? It's fragmentation of primary key (and whole data table, as InnoDB data == primary key). But how it could happen with
primary key after mysqldump ? The answer here if we look on

EXPLAIN SELECT * FROM tracker;

We see that dump is taken in key "block_id" order, not in primary key order. And later when we load this table, INSERTS into primary key happens in random order, and that gives us the fragmentation we see here.

How to fix it in our case. It's easy: ALTER TABLE tracker ENGINE=InnoDB, it will force InnoDB to rebuild table in primary key order.

After that Q1:

You see that time returned to appropriate 17.72 sec.

You may ask what happens now with Q2 ? yes, it's getting slow now, as we made key "block_id" inserted not in order.

As for mysqldump you may use --order-by-primary options to force dump in primary key order.

So notes to highlight:

• InnoDB fragmentation may hurt your query significantly, especially when data is not in buffer_pool and execution goes to read from disk
• Fragmentation by secondary key is much more likely than by primary key, and you cannot really control it (tough it is possible in XtraDB / InnoDB-plugin with FAST INDEX creation) so be careful with queries scan many records by secondary key
• To check if you query affected by fragmentation you can use Innodb_scan_pages_contiguous ; Innodb_scan_pages_jumpy statistics in 5.0-percona releases (coming to 5.1-XtraDB soon)

Entry posted by Vadim | 7 comments

Add to: | | | |

I said in my last post, that we're interested in speaking at MySQL meetups, and I'm happy to say that the Los Angeles MySQL Meetup has taken us up on the offer.

On November 18th, I'll be giving an introductory talk on InnoDB/XtraDB Performance Optimization.  I will be the second speaker, with Carl Gelbart first speaking on Infobright.

What brings me to LA?  On the same day (18th Nov) I'll be teaching a one day class on Performance Optimization for MySQL with InnoDB and XtraDB.  If you haven't signed up yet - spaces are still available.

Entry posted by Morgan Tocker | No comment

Add to: | | | |

So long time after PPC 2009 there was available version mysql-galera-0.6, which had serious flow, to setup a new node you had to take down whole cluster. And all this time Codership ( company that develops Galera) was working on 0.7 release that introduces node propagation keeping cluster online. You can play with 0.7pre release by yourself MySQL/Galera Release 0.7pre.

In current version propagation is done by mysqldump from one of nodes ("donor"). In next release Codership is going to support LVM snapshot and xtrabackup which will make the setup of new node even easier. The current annoyance I see is that if you shutdown one node for short period of time for quick maintenance, after start, the node has to load whole mysqldump, like it is new empty node. I hope Codership guys will address this also.
Another thing I miss for now is support of InnoDB-plugin, which as we know performs much better than standard InnoDB ®.

So what is so interesting about Galera. Couple things:

- High Availability. Any of N standby nodes are available immediately when main node fails. Galera is serious pretender to be included to the list, Yves put recently, http://www.mysqlperformanceblog.com/2009/10/16/finding-your-mysql-high-availability-solution-%e2%80%93-the-questions/. I am not sure how many nines it will provide , but efforts on test setup and deployment should be comparable with MMM setup.

- Scale Writes. Galera allows to write to any of N nodes and automatically propagate to other nodes. It sounds too ideal, and there is drawback - with increasing amount of nodes you write to, your transaction rollback rate may increase, especially if you working on the same dataset. You can find some results on Codership's page, and I am going to run my own benchmarks also. Also from benchmark you can see that communication overhead maybe significant for short writes.

- Scale Reads. It can be done with regular replication, but with synchronous your "slaves-nodes" are in the same state, there is no "slave behind". When you read from any slave, you read actual data. Although it also has serious drawback - our cluster is fast as fast the "weakest" node in the chain. So if one node gets overloaded and performance degrades, the same happens with whole cluster.

- Heterogeneous-database replication. It is not here yet, and I do not know what's in Codership roadmap, but group manager protocol in Galera is database independent, and it's only matter of database drivers. For InnoDB currently it is set of patches, and I see it is quite possible to make the same for Postgres. So MySQL-Postgres cluster setup is not so far ahead

On "Company page" Codership says their goal is "to promote and exploit the latest developments in computer science to produce fast and scalable synchronous replication solution that "just works" for databases and similar applications", which I think they have success in. Implementing fast, scalable and working group communication and transaction manager is the art.

As for now I would not put 0.7 release into production yet, but you may seriously consider to play with it in test environment, and report bugs to Codership team, they are very responsive.
I am waiting for next releases and looking to make integration with XtraDB.

Entry posted by Vadim | 6 comments

Add to: | | | |

At first, <tuning peak performance>. The next, <tuning purge operation> to stabilize performance  and to avoid decreasing performance.

<test condition>

Server:
PowerEdge R900, Four Quad Core E7320 Xeon, 2.13GHz, 32GB Memory, 16X2GB, 667MHz

db_STRESS:
32 sessions, RW=1, dbsize = 1000000, no thinktime

XtraDB: (mysql-5.1.39 + XtraDB-1.0.4-current)
innodb_io_capacity = 4000
innodb_support_xa = false
innodb_file_per_table = true
innodb_buffer_pool_size = 16G
innodb_read_io_threads = 8
innodb_write_io_threads = 8
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 128M
innodb_log_file_size = 512M
innodb_log_files_in_group = 2
innodb_max_dirty_pages_pct = 90
innodb_flush_method = O_DIRECT
(the followings are XtraDB specific general settings)
innodb_ibuf_active_contract = 1
innodb_adaptive_flushing = false
innodb_adaptive_checkpoint = estimate

<tuning peak performance>

At first, tuning the peak performance to use CPU and IO resource more effectively. To avoid mutex/lock contentions are good to use more CPU resource of many CPUs.

This graph shows the peak performance in tps of db_STRESS.

At current settings, "base" in the graph is the perfomance. We can confirm the mutex/lock contention roughly by the SEMAPHORES sction of SHOW INNODB STATUS output.

"xx-lock on RW-latch at 0x7f2ff40a3dc0 created in file dict/dict0dict.c line 1627"

It is index->lock, viewing the source file (and it may be HISTORY table). This is the lock for each index tree. We may be able to disperse the lock using by the partitioning of MySQL. Added the following clause to the HISTORY table definition.

"PARTITION BY HASH(REF_OBJECT) PARTITIONS 16"

Now the performance became to "+partitioned" in the graph. Looking the SEMAPHORES section again,

"has waited at handler/ha_innodb.cc line 7275 for 0.0000 seconds the semaphore:
X-lock on RW-latch at 0xd30320 created in file dict/dict0dict.c line 623"

may be the line which appears for the most times (it is dict_operation_lock). It may be partition specific lock contention. The current XtraDB has the variable to tune the contention.

innodb_stats_update_need_lock = 0 (default 1)

It skip the updating statistics which needs the lock. (it only affects for "Data_free:" value of TABLE STATS). And the performance became "+skip_stats" in the graph.  Then, the next contention at SEMAPHORES section is...

"Mutex at 0x1b3e3e78 created file trx/trx0rseg.c line 167"

may be remarkable (it is rseg->mutex). The mutex is for each rollback segments, so we can increase the rsegs to solve the contention problem. XtraDB can increase the rseg.

innodb_extra_rsegments = 64 (affects to initialization of InnoDB)

Recreated database files with the parameter. Then the performance became "+rsegs64". At last, the next contention may be "Mutex at 0x28ce8e0 created file srv/srv0srv.c line 982". It is kernel_mutex, currently we don't have proper solution for that. The setting seems to be enough for now.

<tuning purge operation>

Next, looking the sequential result in more long term.

The next problem is "History list length" growing to huge size. The value is the number of entries in rollback segment. The entries are used for consistent reading of the older transactions. They can be removed when any transactions doesn't refer the entry. This removing operation for the entries is called "purge" in InnoDB. The purge operation should be done enough on time, because the huge history list affects to performance.

Basically, the purging is done by master_thread (general background thread of InnoDB). The huge history list makes the purge operation slow, and it interferes  with the other tasks of the master_thread (e.g. flushing dirty blocks, treating insert buffer, etc...). Dimitri implemented a purge_thread to devote to the purging, and also XtraDB has similar purge_thread. Though it seems to make the throughput stabilize, it is not enough still for heavy update workloads. A single purge_thread on one CPU is not enough for updates from user threads on the all of other CPUs.

XtraDB can increase the purge_threads from the next release.

innodb_use_purge_thread = 4

seems to be enough for this workload on the server.

The first graph of followings is sequential throughput [tps] up to 3500 sec.

The next is tracking the "History list length" at the same time.

"Norm 1.0.4": Normal InnoDB Plugin 1.0.4 without XtraDB specific options
"xtra p_t 0": XtraDB 1.0.4-new (no purge_thread)
"xtra p_t 1": XtraDB 1.0.4-new (single purge_thread similar to Dimitri's)
"xtra p_t 4": XtraDB 1.0.4-new (4 purge threads)

The graphs show...

• The purge thread (> 0) helps to stabilize the throughput greatly.
• Increasing the purge threads can suppress the strong growing of the hitory list
• The adaptive checkpoint "estimate" needs the purge_thread... (than the adaptive_flushing does)

And the last 300secs' average tps are...

"Norm 1.0.4": 5725.47
"xtra p_t 0": 4699.33
"xtra p_t 1": 7130.3
"xtra p_t 4": 9118    (about 60%up from Normal Plugin 1.0.4)

In the end, the faster and more stable performance of db_STRESS benchmark is obtained by these tunings of XtraDB.

-----------------------------------------

(Added 2009.10.29)

<FAQ: Is XtraDB slower than Plugin?>

I'd like to say "no" to this question. We have been adding many tuning options to XtraDB. But they are effective not for all cases, sometimes the performance may get worse because of "not proper" or "too much value". We should choose the options correctly. XtraDB is based on InnoDB Plugin and we can set XtraDB same to InnoDB Plugin at least. The following graphs are results of XtraDB and Plugin with same options and same database.

top-left (same condition to above graphs):
innodb_flush_log_at_trx_commit = 2
innodb_doublewrite = true

top-right:
innodb_flush_log_at_trx_commit = 1
innodb_doublewrite = true

bottom-left:
innodb_flush_log_at_trx_commit = 2
innodb_doublewrite = false

bottom-right:
innodb_flush_log_at_trx_commit = 1
innodb_doublewrite = false

It seems that XtraDB is not slower than Plugin here at least.

We can start tuning based on these performances using XtraDB specific options!

Why do you make XtraDB slower than Plugin?

Entry posted by Yasufumi | 9 comments

Add to: | | | |

More specially:

I populated each of the tables with 12M rows. Getting 7 bytes row size for first table and 101 byte for second and third.
I used simple scan query: select max(t1) from t100; for the test.

The result was as follows:
t1 - 1.00 sec (12M rows/sec ; 80MB/sec)
t1c99 - 1.71 sec (7M rows/sec ; 676MB/sec)
t100 - 1.77 sec (7M rows/sec ; 653MB/sec)
t99v1 - 12.36 sec (1M rows/sec ; 93MB/sec)

This shows there is surely the problem with dynamic row format table with many columns. But is it because of large number of columns or dynamic format on its own is slave ?
I have tested yet another table structure:

This table has row length of 20 (which was a bit of surprise to me) and it has:

t1v1 - 1.83 sec (6.5M rows/sec; 125M/sec)

So there is surely the penalty for dynamic rows, however it is not very significant if number of columns is small. For large number of columns dynamic rows become very expensive and you have to watch out.
I have not looked at the code and would appreciate any developers comments but I guess for dynamic rows tables certain conversion has to take place when internal data structures are populated (everything but TEXTs/BLOBs is fixed length when it is being processed). This conversion process depends on number of columns while for fixed rows the MyISAM storage format matches internal one so you can basically do memory copy which does not depends on number of columns.

Another interesting observation is access speed to different columns. the max(t1) and max(t99) were taking the same time which means there is no penalty for accessing column which is in the end of the table rather than at the start when it comes to MyISAM.

The common workaround working with such wide tables is to use covering indexes. I added one to t99v1 table and repeated the query:

As you can see the index scan is not as fast as table scan scanning about 3.7M rows/sec but which is still pretty fast.

So this is all about MyISAM, what is about Innodb ? Here are results for Innodb with all data in buffer pool, to measure peak speed as well

The results for Innodb were:
t1 - 5.11 sec (2.3M rows/sec)
t1c99 - 5.74 sec (2.1M rows/sec)
t100 - 15.16 sec (0.8M rows/sec)
t99v1 - 14.93 sec (0.8M rows/sec)
t1v1 - 5.26 sec (2.3M rows/sec)
t99v1 (covering idx) - 5.62 sec (2.1M rows/sec)

As you can see Innodb is a lot slower and has behavior similar to Dynamic Row tables in both cases. This is because Innodb does not store data in native MyISAM format and conversion is needed in all cases. We can also see the table scan speed can be up to 5 times slower, for very short rows - some of this goes back to the fact Innodb rows have a lot of transaction control overhead attached to them.

Also note the covering index scan speed is very similar to full table scan speed - this is rather expected as table data is stored in BTREE index very similarly to how indexes are stored.

Summary: Beware of dynamic row format tables with many columns they might bite you with surprise slowdown. MyISAM is much faster than Innodb when it comes to in memory full table scan.

P.S Tests were done on MySQL 5.4.2 on Intel(R) Xeon(R) CPU E5405 @ 2.00GHz CPU.

Entry posted by peter | 9 comments

Add to: | | | |

Here I'll grab a bunch of cardinality estimates from sakila.film on MySQL 5.0.45 and put them into a file:

After a while I cancel it and then sort and aggregate them with counts:

Look at the distribution of the counts. The weighted average of these is 1000.53, so it's close to the truth (1000 rows). But five of the eight distinct estimates are shown about one-half as often as the others; it looks like the random choice of which statistic to use is not evenly distributed.

I mentioned this to Heikki and he pondered it for a bit -- but neither of us really figured out what was going on. I know the code superficially, but not as well as he or Yasufumi or others do; and I was not able to find a cause.

More recently I saw that I'm not the only one who notices oddities in the random number generation. I waited. And indeed the fixes for that bug seemed to have fixed the skew in the statistics. Case solved, and all I had to do was wait. Truly, laziness is a virtue.

Entry posted by Baron Schwartz | 5 comments

Add to: | | | |

In the case of heavy updating workload, the modified ages of the many blocks are clustered. And to reduce the maximum of the modified ages InnoDB needs to flush many of the blocks in a short time, if these are not flushed at all. Then the flushing storm affect the performance seriously.

We suggested the "adaptive_checkpoint" option of constant flushing to avoid such a flushing storm. And finally, the newest InnoDB Plugin 1.0.4 has the new similar option "adaptive_flushing" as native.

Let's check the adaptive flushing options at this post.

HOW THEY WORK

< adaptive_checkpoint=reflex (older method)>

< adaptive_checkpoint=estimate (newer method)>

* estimate blocks to flush based on...

• how many modified and unflushed blocks
• progress speed of the transaction log
• the modified age average of all blocks to flush

< adaptive_flushing (default "true" at 1.0.4)>
Its behavior is not based on the oldest modified age %.

* the how many blocks to flush based on...

• how many modified and unflushed blocks
• progress speed of the transaction log
• transaction log capacity

(and adjust along with another flushing)

And it doesn't exceed the equivalent to "constant flushing (strong)"
of the "adaptive_checkpoint=reflex"

RESULTS

TPC-C like workload (100WH: 16 session full)

innodb_buffer_pool_size = 16G
innodb_max_dirty_pages_pct = 90

innodb_log_file_size = 512M
innodb_log_files_in_group = 2

innodb_io_capacity = 4000
innodb_read_io_threads = 8
innodb_write_io_threads = 8
innodb_flush_method = O_DIRECT

innodb_thread_concurrency = 0
innodb_ibuf_active_contract = 1

<none>
innodb_adaptive_flushing = false
innodb_adaptive_checkpoint = none

<flushing>
innodb_adaptive_flushing = true

<reflex>
innodb_adaptive_flushing = false
innodb_adaptive_checkpoint = reflex

<estimate>
innodb_adaptive_flushing = false
innodb_adaptive_checkpoint = estimate

case1: "innodb_doublewrite = false"

[0~1800 sec.]

[1200~1800 sec.]

averages (1200~1800 sec.)
none:     6868.92
flushing: 6655.92
reflex:     6481
estimate: 6575.88

case2: "innodb_doublewrite = true"

[0~1800 sec.]

[1200~1800 sec.]

averages (1200~1800 sec.)
none:     6569.48
flushing: 5090.12
reflex:     6871.9
estimate: 6609.9

CONSIDERINGS

The new "adaptive_flushing" seems to be tuned for "innodb_doublewrite = false" only.
(cause too much flushing for "innodb_doublewrite = true")

"innodb_adaptive_checkpoint = reflex" and "adaptive_flushing" need tuning innodb_io_capacity properly.
(The result is based on proper value "innodb_io_capacity = 4000")

"innodb_adaptive_checkpoint = estimate" is not depend on innodb_io_capacity, and it seems more "soft" result than the other methods.

So,

The "adaptive_flushing" seems good when we use "innodb_doublewrite = false".

"innodb_adaptive_checkpoint = estimate" may be safe for "innodb_doublewrite = true" for now.

I should adjust "adaptive_flushing" for "innodb_doublewrite = true".

I have tried to improve the performance. But I couldn't adjust performance of adaptive_flushing with doublewrite... So, instead of that, estimate is adjusted more.

case1: "innodb_doublewrite = false"

[0~1800 sec.]

case2: "innodb_doublewrite = true"

[0~1800 sec.]

The new "innodb_adaptive_checkpoint = estimate" seems to be more stable than before. The next question is, "Why is estimate + doublewrite better than the other no doublewrite settings?"...

Also tested in another configuration to check the behaviors in another balance of IO bound.

<changes>
innodb_buffer_pool_size = 2G   (1/8)
innodb_log_file_size = 128M    (1/4)

case1: "innodb_doublewrite = false"

[0~1200 sec.]

case2: "innodb_doublewrite = true"

[0~1200 sec.]

"innodb_adaptive_checkpoint = estimate" seems stable but seems to flush much in its initial phase. The adaptive_flushing seems to be more soft in this cases.

Hmmm....

We should choose suitable adaptive_xxx for each workload for now...

Entry posted by Yasufumi | 12 comments

Add to: | | | |

--------------
ROW OPERATIONS
--------------
8 queries inside InnoDB, 9 queries in queue
100 read views open inside InnoDB

It is relationship between queries active - queries inside innodb+queries in the queue totalling 17 with "read views open inside InnoDB" which is a fancy way of saying "active transactions" which is 100.

Typically you would want this ratio to be close to 1, may be 2 which would correspond all active transactions doing active work and spending little time waiting on the application. Waiting transactions are bad because they hold lock as well as other resources, such as preventing innodb purge operation from proceeding.

If you see something like this in your envinronment it often makes sense to check the the list of transactions too. Chances are you would see something like:

---TRANSACTION 0 1044183147, ACTIVE 963 sec, process no 27713, OS thread id 1227987264 waiting in InnoDB queue

which corresponds to transaction active for over 15 minutes.

Sometimes you would see some query associated with such transactions showing up often in the processlist, which often corresponds to some long batch jobs, sometimes you would just see the processlist showing connection is in "Sleep" mode for a long time which means application is holding transaction open while waiting for something else. Most of such situations are bugs either in application itself or connection pooling software.

Entry posted by peter | No comment

Add to: | | | |

We chose to put this into xtrabackup for a couple of reasons - the first is that running statistics on your backup database does not need to hurt production servers, and the second reason is that running statistic on a stopped database is more accurate than with online (although online is also supported, but you may have inexact results).

Let's see how it works. I have one table with size 13Gb what was filled during about 2.5 years.
The table is:

And size of file is about 12.88 GB

-rw-r--r-- 1 root root 13832814592 Sep 10 14:41 link_out104.ibd


So to get statistics we run:

xtrabackup --stats --tables=art.link* --datadir=/mnt/data/mysql/

which will show something like this:

The output is intensive, let me highlight some points:

It says that PRIMARY key (which is the table by itself, as InnoDB is clustering data by primary key) takes 497839 pages ( 16KB each) and size of data 7492026403 bytes or (6.98 GB). And density ( fitting data into pages) is quite good - 91%. But it was expected, as table is really mostly inserted in, updates and deletes are rare).

And let's take index domain_id

you can see the allocated pages (43255 pages or 708689920 bytes) are filled only by 76% ( data takes 545031333 bytes). And that means that 150MB are just waste of space. Which is really even worse for key revert_domain

leaf pages: recs=25839414, pages=133869, data=1566961607 bytes, data/pages=71%
.

For this key about 600MB is empty.

This needs a bit of explaining:
This does not have as good efficiency as the primary key, but a lot of this is to be expected. In a lot of cases we insert into the primary key in order which makes things very predictable, but the inserts into the secondary key index are random - which leads to a lot of page splits.

One helpful new feature to address this is in XtraDB/InnoDB plugin - fast index creation. With this feature, InnoDB creates indexes by sort, so page fill factor should be quite good.

To check that, there is xtrabackup --stats for index domain_id created for table in Barracuda format with Fast creation method:

As you see this time it takes 34383 pages (compare to 43255 in previous statistics).

Though it would be interesting to see how it will grow with further inserts, and I also suspect random INSERTS into so dense space going to be slower than in previous case.

The --stats is not in xtrabackup release yet, only in source code repository, but should be released quite soon.

And the last point of the post - if you are badly missing some features in MySQL, InnoDB, InnoDB-plugin, XtraDB, XtraBackup - you know whom ask for!

Entry posted by Vadim | 5 comments

Add to: | | | |