Understanding how well your tables and indexes fit to buffer pool are often very helpful to understand why some queries are IO bound and others not – it may be because the tables and indexes they are accessing are not in cache, for example being washed away by other queries. MySQL Server does not provide any information of this type, Percona Server however adds number of tables to Information Schema which makes this information possible. It is just few queries away:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 | SELECT `schema` AS table_schema, innodb_sys_tables.name AS table_name, innodb_sys_indexes.name AS index_name, cnt, dirty, hashed, ROUND(cnt * 100 / index_size, 2) fit_pct FROM (SELECT index_id, COUNT(*) cnt, SUM(dirty = 1) dirty, SUM(hashed = 1) hashed FROM innodb_buffer_pool_pages_index GROUP BY index_id) bp JOIN innodb_sys_indexes ON id = index_id JOIN innodb_sys_tables ON table_id = innodb_sys_tables.id JOIN innodb_index_stats ON innodb_index_stats.table_name = innodb_sys_tables.name AND innodb_sys_indexes.name = innodb_index_stats.index_name AND innodb_index_stats.table_schema = innodb_sys_tables.SCHEMA ORDER BY cnt DESC LIMIT 20; +--------------+--------------+--------------+------+-------+--------+---------+ | table_schema | table_name | index_name | cnt | dirty | hashed | fit_pct | +--------------+--------------+--------------+------+-------+--------+---------+ | test | a | c | 7976 | 0 | 0 | 13.73 | | test | a | PRIMARY | 59 | 0 | 0 | 0.08 | | sbtest | sbtest#P#p1 | PRIMARY | 22 | 0 | 0 | 22.68 | | sbtest | sbtest#P#p0 | PRIMARY | 22 | 0 | 0 | 22.68 | | sbtest | sbtest#P#p2 | PRIMARY | 21 | 0 | 0 | 21.65 | | sbtest | sbtest#P#p3 | PRIMARY | 18 | 0 | 0 | 18.56 | | sbtest | sbtest#P#p3 | k | 4 | 0 | 0 | 100.00 | | sbtest | sbtest#P#p2 | k | 4 | 0 | 0 | 100.00 | | sbtest | sbtest#P#p1 | k | 4 | 0 | 0 | 100.00 | | sbtest | sbtest#P#p0 | k | 4 | 0 | 0 | 100.00 | | stats | tables | PRIMARY | 2 | 0 | 0 | 66.67 | | stats | tables | TABLE_SCHEMA | 1 | 0 | 0 | 100.00 | | percona | transactions | PRIMARY | 1 | 0 | 0 | 100.00 | +--------------+--------------+--------------+------+-------+--------+---------+ 13 rows in set (0.04 sec) |
This query shows information about how many pages are in buffer pool for given table (cnt), how many of them are dirty (dirty), and what is the percentage of index fits in memory (fit_pct)
For illustration purposes I’ve created one table with partitions to show you will have the real “physical” table name which identifies table down to partition, which is very helpful for analyzes of your
access to partitions – you can actually check if your “hot” partitions really end up in the cache and “cold” are out of the cache, or is something happening which pushes them away from the cache.
You can use this feature to tune buffer pool invalidation strategy, for example play with innodb_old_blocks_pct and innodb_old_blocks_time actually observing data stored in buffer pool rather than using some form of temporary measures.
I often check these stats during warmup to see what is really getting warmed up first as well as how buffer pool is affected by batch jobs, alter tables, optimize table etc – the lasting impact these may have on system performance is often caused by impact they have on buffer pool which may take hours to recover.
This tool can be also helpful for capacity planning/performance management. In many cases you would learn you need a certain fit to buffer pool for tables/indexes for reasonable performance, you may try to count it too but it may be pretty hard as there are a lot of variables, including page fill factors etc.
Hey I am using percona server 5.1.43. I tried to execute the above query with “information_schema” database selected but errors out with
ERROR 1109 (42S02): Unknown table ‘innodb_sys_indexes’ in information_schema
Looks like innodb_sys_indexes is available only in latest version on percona server.
Yashh,
This feature was added later. Please try latest stable version
Hi,
Is it possible to get some (or at least something relevant) of this information from MySQL itself not from Percona server?
Thanks,
Paulo
Paulo, no. Standard MySQL/InnoDB is completely un-measurable in this regard.
Small correction of query to support 5.5.13
SELECT
schemaAS table_schema,innodb_sys_tables.name AS table_name,
innodb_sys_indexes.name AS index_name,
cnt,
dirty,
hashed,
ROUND(cnt * 100 / index_size, 2) fit_pct
FROM (SELECT index_id,
COUNT(*) cnt,
SUM(dirty = 1) dirty,
SUM(hashed = 1) hashed,
data_size index_size
FROM innodb_buffer_pool_pages_index
GROUP BY index_id) bp
JOIN innodb_sys_indexes
ON bp.index_id = innodb_sys_indexes.index_id
JOIN innodb_sys_tables
ON innodb_sys_indexes.table_id = innodb_sys_tables.table_id
JOIN innodb_index_stats
ON innodb_index_stats.table_name = innodb_sys_tables.name
AND innodb_sys_indexes.name = innodb_index_stats.index_name
AND innodb_index_stats.table_schema = innodb_sys_tables.SCHEMA
ORDER BY cnt DESC
LIMIT 20;
I have the same problem as Steve – Unknown column ‘index_size’ when running on Percona 5.5.12
And I meant information_schema not performance_schema of course…
Sorry that should be index_size
Hi Peter. I tried this on 5.5.11.
Get ERROR 1054 (42S22): Unknown column ‘index_length’ in ‘field list’
ran mysql_upgrade –force to fix any differences in performance_schema
Still no joy.
This is a binary distro that I have configured to run with data files created on MySQL 5.5.11 mainline….
Error when using this request on MariaDB 5.2.5
110515 16:43:36 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=0
read_buffer_size=524288
max_used_connections=281
max_threads=2002
threads_connected=120
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 9249947 K
bytes of memory
Hope that’s ok; if not, decrease some variables in the equation.
Thread pointer: 0x7fd1bccedf00
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong…
stack_bottom = 0x7fd1c1b6f0b8 thread_stack 0×40000
/usr/sbin/mysqld(my_print_stacktrace+0x2e) [0x7fd959f0b82e]
/usr/sbin/mysqld(handle_segfault+0x3cd) [0x7fd959b4c89d]
/lib/libpthread.so.0 [0x7fd95815a190]
/usr/sbin/mysqld [0x7fd959e03d43]
/usr/sbin/mysqld(get_schema_tables_result(JOIN*, enum_schema_table_state)+0×211) [0x7fd959c64d41]
/usr/sbin/mysqld(JOIN::exec()+0x4dd) [0x7fd959bca8ed]
/usr/sbin/mysqld(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*)+0x12a) [0x7fd959bc6b6a]
/usr/sbin/mysqld(handle_select(THD*, st_lex*, select_result*, unsigned long)+0x15d) [0x7fd959bcc81d]
/usr/sbin/mysqld [0x7fd959b5749a]
/usr/sbin/mysqld(mysql_execute_command(THD*)+0x40e) [0x7fd959b5afae]
/usr/sbin/mysqld(mysql_parse(THD*, char*, unsigned int, char const**)+0x2c1) [0x7fd959b606e1]
/usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x5b1) [0x7fd959b60ca1]
/usr/sbin/mysqld(do_command(THD*)+0xf4) [0x7fd959b61ed4]
/usr/sbin/mysqld(handle_one_connection+0x13e) [0x7fd959b5376e]
/lib/libpthread.so.0 [0x7fd958151a04]
/lib/libc.so.6(clone+0x6d) [0x7fd95771ed4d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7fd1bccf90b8): is an invalid pointer
Connection ID (thread ID): 495168
Status: NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
110515 16:43:39 mysqld_safe Number of processes running now: 0
110515 16:43:39 mysqld_safe mysqld restarted
Has this command changed?
I am new to Percona and using 5.5.20-rel24.1.
I tried Alex S updated query for Percona-Server-shared-55-5.5.14 and I was getting fit_pct values that were greater than 100%. I also tried Peter’s query and I got the same “Unknown column ‘index_size’” that other people were seeing. Has this been updated at all?
This post is highly ranked, but the query doesn’t even work. Can we get an updated query?
Vadim,
As always, excellent info.
From your post it would appear this is percona-specific. Is this proprietary, or is there a chance of those table statistics making their way into mysql / mariadb?
Up vote for Mike Purcell suggestion to update this post!
running percona 5.6
What is the equivalent of this in 5.6?