Hosting a shared MySQL instance for your internal or external clients (“multi-tenant”) was always a challenge. Multi-tenants approach or a “schema-per-customer” approach is pretty common nowadays to host multiple clients on the same MySQL sever. One of issues of this approach, however, is the lack of visibility: it is hard to tell how many resources (queries, disk, cpu, etc) each user will use.
Percona Server contains userstats Google patch, which will allow you to get the resource utilization per user. The new MySQL 5.6 performance_schema has even more instrumentation which can give you a better visibility on per-user or per-schema/per-database level. And if you are running MySQL 5.6.6 or higher, Performance Schema is already enabled (with minimum set of “instrumentation” thou) and ready to use. In this post I will share examples of using Performance Schema for multi-tenant environments.
Overview
If you want to use Performance Schema, make sure it is enabled (enabled by default starting with MySQL 5.6.6):
1 2 3 4 5 6 7 | mysql> show global variables like 'performance_schema'; +--------------------+-------+ | Variable_name | Value | +--------------------+-------+ | performance_schema | ON | +--------------------+-------+ 1 row in set (0.00 sec) |
Performance_schema provides you with the “raw” metrics and it may be difficult to select data from it. The good news is that you can use the “sys” schema project by Mark Leith. The “sys” schema (old name: ps_helper) is a collection of views and stored procedures which will provide you with reports in human readable format. The installation is easy, download it from github and run:
1 | $ mysql -u root -p < ./sys_56.sql |
(it will only create database “sys” and a set of views and stored procedures/stored functions in it)
Examples
For the multi-tenant environment the most interesting is resource utilization breakdown. Lets say you want to “charge per IO” similar to Amazon RDS for MySQL model. You can now run this simple query against “sys” schema, which will use Performance Schema in MySQL 5.6 to generate report (in my example all users starts with “a”):
1 2 3 4 5 6 7 8 9 10 11 12 | mysql sys> select * from user_summary_by_file_io where user like 'a%'; +-------+------------+-------------+ | user | ios | io_latency | +-------+------------+-------------+ | api01 | 3061469563 | 4.04h | | api03 | 912296937 | 1.36h | | api02 | 815473183 | 1.22h | | app01 | 9704381 | 00:01:06.33 | | app02 | 1160149 | 8.18 s | | app03 | 1424065 | 7.23 s | +-------+------------+-------------+ 6 rows in set (0.06 sec) |
If you need more extensive metrics you can use this report:
1 2 3 4 5 6 7 8 9 10 11 12 | mysql sys> select * from user_summary where user like 'a%' limit 10; +-------+------------+-------------------+-----------------------+-------------+------------+-----------------+---------------------+-------------------+--------------+ | user | statements | statement_latency | statement_avg_latency | table_scans | file_ios | file_io_latency | current_connections | total_connections | unique_hosts | +-------+------------+-------------------+-----------------------+-------------+------------+-----------------+---------------------+-------------------+--------------+ | api01 | 1837898366 | 192.29h | 376.65 us | 2221018 | 3064724488 | 4.05h | 22 | 73748519 | 1 | | api02 | 833712609 | 206.83h | 893.11 us | 1164 | 836753563 | 1.26h | 7 | 246730 | 1 | | api03 | 988875004 | 222.03h | 808.29 us | 1156 | 943363030 | 1.43h | 8 | 247639 | 1 | | app01 | 1864159 | 4.01h | 7.75 ms | 46756 | 9704623 | 00:01:06.33 | 0 | 300004 | 1 | | app02 | 812520 | 00:20:51.69 | 1.54 ms | 42513 | 1164954 | 8.37 s | 0 | 122994 | 1 | | app03 | 811558 | 00:21:59.57 | 1.63 ms | 42310 | 1429373 | 7.40 s | 0 | 123110 | 1 | +-------+------------+-------------------+-----------------------+-------------+------------+-----------------+---------------------+-------------------+--------------+ 6 rows in set |
Or a breakdown per user per statement:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | mysql sys> select user, statement, total, rows_sent, rows_examined, rows_affected, full_scans from user_summary_by_statement_type where user = 'api01' limit 100; +-------+-------------+-----------+-----------+---------------+---------------+------------+ | user | statement | total | rows_sent | rows_examined | rows_affected | full_scans | +-------+-------------+-----------+-----------+---------------+---------------+------------+ | api01 | insert | 368405774 | 0 | 0 | 494179199 | 0 | | api01 | commit | 376276686 | 0 | 0 | 0 | 0 | | api01 | select | 476129532 | 761038004 | 1067363348 | 0 | 0 | | api01 | update | 142896430 | 0 | 142896437 | 141978066 | 0 | | api01 | begin | 376280424 | 0 | 0 | 0 | 0 | | api01 | show_fields | 2215016 | 310102240 | 310102240 | 0 | 2215016 | | api01 | Quit | 73421361 | 0 | 0 | 0 | 0 | | api01 | change_db | 17258767 | 0 | 0 | 0 | 0 | | api01 | delete | 7896 | 0 | 7896 | 7896 | 0 | | api01 | rollback | 3711 | 0 | 0 | 0 | 0 | +-------+-------------+-----------+-----------+---------------+---------------+------------+ 10 rows in set (0.04 sec) |
If you are using “schema per customer” approach you get get the per-schema resource utilization using sys.schema_table_statistics. Example (from Peter Zaitsev’s webinar):
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | mysql [localhost] {msandbox} > select * from schema_table_statistics where table_schema='sbtest' limit 1 G *************************** 1. row *************************** table_schema: sbtest table_name: sbtest rows_fetched: 158764154 fetch_latency: 1.37h rows_inserted: 378901 insert_latency: 00:07:17.38 rows_updated: 1136714 update_latency: 00:45:40.08 rows_deleted: 378902 delete_latency: 00:03:00.34 io_read_requests: 636003 io_read: 9.70 GiB io_read_latency: 00:28:12.01 io_write_requests: 203925 io_write: 3.11 GiB io_write_latency: 17.26 s io_misc_requests: 2449 io_misc_latency: 3.87 s |
This report may be really slow if you have lots of tables. If you are only interested in disk utilization per database you can directly query the performance_schema:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | SELECT extract_schema_from_file_name(fsbi.file_name) AS table_schema, SUM(fsbi.count_read) AS io_read_requests, sys.format_bytes(SUM(fsbi.sum_number_of_bytes_read)) AS io_read, sys.format_time(SUM(fsbi.sum_timer_read)) AS io_read_latency, SUM(fsbi.count_write) AS io_write_requests, sys.format_bytes(SUM(fsbi.sum_number_of_bytes_write)) AS io_write, sys.format_time(SUM(fsbi.sum_timer_write)) AS io_write_latency, SUM(fsbi.count_misc) AS io_misc_requests, sys.format_time(SUM(fsbi.sum_timer_misc)) AS io_misc_latency FROM performance_schema.file_summary_by_instance AS fsbi GROUP BY table_schema ORDER BY io_write_requests DESC limit 10; +--------------------+------------------+------------+-----------------+-------------------+------------+------------------+------------------+-----------------+ | table_schema | io_read_requests | io_read | io_read_latency | io_write_requests | io_write | io_write_latency | io_misc_requests | io_misc_latency | +--------------------+------------------+------------+-----------------+-------------------+------------+------------------+------------------+-----------------+ | binlogs | 6943921992 | 19.57 TiB | 2.46h | 1124408959 | 6.60 TiB | 2.18h | 914411 | 00:03:55.75 | | d1 | 146448 | 2.24 GiB | 1.97 s | 22768241 | 241.44 GiB | 00:12:57.97 | 5095817 | 00:12:50.16 | | stats1 | 853 | 1.81 MiB | 97.12 ms | 1042768 | 15.93 GiB | 10.27 s | 292303 | 22.74 s | | stats2 | 3024 | 11.22 MiB | 781.96 ms | 750182 | 11.45 GiB | 9.32 s | 267780 | 50.84 s | | relaylogs | 105065 | 925.71 MiB | 312.82 ms | 447607 | 925.71 MiB | 1.19 s | 72169 | 530.70 ms | | o1268 | 2078 | 3.02 MiB | 7.40 ms | 1591 | 203.97 KiB | 1.17 ms | 2693 | 92.34 ms | +--------------------+------------------+------------+-----------------+-------------------+------------+------------------+------------------+-----------------+ |
Edit: if you also need to get a quick “hot tables” report with focus on disk utilization (io requests) you can use this query:
1 2 3 4 5 6 7 8 9 10 11 12 13 | SELECT concat(extract_schema_from_file_name(fsbi.file_name), '.', extract_table_from_file_name(fsbi.file_name) ) as unique_table_name, SUM(fsbi.count_read) AS io_read_requests, sys.format_bytes(SUM(fsbi.sum_number_of_bytes_read)) AS io_read, sys.format_time(SUM(fsbi.sum_timer_read)) AS io_read_latency, SUM(fsbi.count_write) AS io_write_requests, sys.format_bytes(SUM(fsbi.sum_number_of_bytes_write)) AS io_write, sys.format_time(SUM(fsbi.sum_timer_write)) AS io_write_latency, SUM(fsbi.count_misc) AS io_misc_requests, sys.format_time(SUM(fsbi.sum_timer_misc)) AS io_misc_latency FROM performance_schema.file_summary_by_instance AS fsbi GROUP BY unique_table_name ORDER BY io_write_requests DESC limit 10; |
This will give you a breakdown per table (and will include system files as well).
Conclusion
Performance_schema in MySQL 5.6 is very useful tool and can help you to get better visualization in MySQL multi-user (multi-tenant) installations. It can also do much more, i.e. find all queries with temporary tables or profile locks and mutex. “sys” schema provide you with the useful pre-defined reports, the table (view) names are almost self explanatory:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | +-----------------------------------------------+ | Tables_in_sys | +-----------------------------------------------+ | innodb_buffer_stats_by_schema | | innodb_buffer_stats_by_table | ... | processlist | ... | schema_tables_with_full_table_scans | | schema_unused_indexes | | statement_analysis | | statements_with_errors_or_warnings | | statements_with_full_table_scans | | statements_with_runtimes_in_95th_percentile | | statements_with_sorting | | statements_with_temp_tables | ... |
If you are using MySQL 5.6, performance_schema will give a additional valuable inside.
