Comments on: MySQL random freezes could be the query cache http://www.mysqlperformanceblog.com/2009/03/19/mysql-random-freezes-could-be-the-query-cache/ Everything about MySQL Performance Sat, 21 Nov 2009 05:23:57 -0800 http://wordpress.org/?v=2.8.4 hourly 1 By: Eric Jensen http://www.mysqlperformanceblog.com/2009/03/19/mysql-random-freezes-could-be-the-query-cache/comment-page-1/#comment-544672 Eric Jensen Mon, 20 Apr 2009 16:39:12 +0000 http://www.mysqlperformanceblog.com/?p=645#comment-544672 This bug turned out to be the query cache too: http://bugs.mysql.com/?id=39508 I was told it is fixed in 5.0.68 This bug turned out to be the query cache too: http://bugs.mysql.com/?id=39508

I was told it is fixed in 5.0.68

]]> By: Noah Everett http://www.mysqlperformanceblog.com/2009/03/19/mysql-random-freezes-could-be-the-query-cache/comment-page-1/#comment-523739 Noah Everett Mon, 30 Mar 2009 00:35:05 +0000 http://www.mysqlperformanceblog.com/?p=645#comment-523739 ah...feel free to delete them, including this one ah…feel free to delete them, including this one

]]> By: Baron Schwartz http://www.mysqlperformanceblog.com/2009/03/19/mysql-random-freezes-could-be-the-query-cache/comment-page-1/#comment-523727 Baron Schwartz Mon, 30 Mar 2009 00:28:53 +0000 http://www.mysqlperformanceblog.com/?p=645#comment-523727 Noah, please don't use the comments as a support forum. We have forums at forums.percona.com and there are also MySQL's mailing list, IRC channel, and forums. And if you need professional help, of course we're here :) Noah, please don’t use the comments as a support forum. We have forums at forums.percona.com and there are also MySQL’s mailing list, IRC channel, and forums. And if you need professional help, of course we’re here :)

]]> By: Noah Everett http://www.mysqlperformanceblog.com/2009/03/19/mysql-random-freezes-could-be-the-query-cache/comment-page-1/#comment-523620 Noah Everett Sun, 29 Mar 2009 23:09:28 +0000 http://www.mysqlperformanceblog.com/?p=645#comment-523620 Ok just caught it when it had froze and had some more info to post: Here is what we get when we try to connect to the server via the command line client: ERROR 1135 (HY000): Can't create a new thread (errno 35); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug Here is what top shows, the mysqld process is at the bottom last pid: 23574; load averages: 0.08, 0.19, 0.33 up 2+01:39:37 23:06:53 1522 processes:1 running, 1521 sleeping CPU: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle Mem: 234M Active, 4999M Inact, 403M Wired, 285M Cache, 214M Buf, 6000K Free Swap: 4096M Total, 4096M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 592 root 1 44 0 24080K 5096K select 0 5:36 0.00% snmpd 630 root 1 44 0 10480K 2136K select 0 0:05 0.00% ntpd 742 root 1 53 0 22876K 2660K select 2 0:04 0.00% sshd 748 root 1 44 0 10696K 2880K select 0 0:02 0.00% sendmail 510 root 1 44 0 5688K 1060K select 2 0:01 0.00% syslogd 758 root 1 8 0 6744K 1184K nanslp 0 0:00 0.00% cron 21290 root 1 44 0 33768K 3500K select 0 0:00 0.00% sshd 752 smmsp 1 20 0 10696K 2704K pause 0 0:00 0.00% sendmail 21294 root 1 20 0 10104K 2284K pause 1 0:00 0.00% csh 20790 mysql 1 8 0 7060K 1388K wait 1 0:00 0.00% sh 23574 root 1 44 0 12208K 5116K CPU2 2 0:00 0.00% top 800 root 1 5 0 5684K 996K ttyin 0 0:00 0.00% getty 799 root 1 5 0 5684K 996K ttyin 0 0:00 0.00% getty 801 root 1 5 0 5684K 996K ttyin 2 0:00 0.00% getty 798 root 1 5 0 5684K 996K ttyin 2 0:00 0.00% getty 803 root 1 5 0 5684K 996K ttyin 2 0:00 0.00% getty 806 root 1 5 0 5684K 944K ttydcd 2 0:00 0.00% getty 797 root 1 5 0 5684K 996K ttyin 1 0:00 0.00% getty 804 root 1 5 0 5684K 996K ttyin 2 0:00 0.00% getty 802 root 1 5 0 5684K 996K ttyin 3 0:00 0.00% getty 805 root 1 5 0 5684K 944K ttydcd 2 0:00 0.00% getty 449 root 1 44 0 2180K 472K select 0 0:00 0.00% devd 20895 mysql 1500 44 0 409M 250M ucond 1 0:00 0.00% mysqld Ok just caught it when it had froze and had some more info to post:

Here is what we get when we try to connect to the server via the command line client:
ERROR 1135 (HY000): Can’t create a new thread (errno 35); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug

Here is what top shows, the mysqld process is at the bottom
last pid: 23574; load averages: 0.08, 0.19, 0.33 up 2+01:39:37 23:06:53
1522 processes:1 running, 1521 sleeping
CPU: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
Mem: 234M Active, 4999M Inact, 403M Wired, 285M Cache, 214M Buf, 6000K Free
Swap: 4096M Total, 4096M Free

PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
592 root 1 44 0 24080K 5096K select 0 5:36 0.00% snmpd
630 root 1 44 0 10480K 2136K select 0 0:05 0.00% ntpd
742 root 1 53 0 22876K 2660K select 2 0:04 0.00% sshd
748 root 1 44 0 10696K 2880K select 0 0:02 0.00% sendmail
510 root 1 44 0 5688K 1060K select 2 0:01 0.00% syslogd
758 root 1 8 0 6744K 1184K nanslp 0 0:00 0.00% cron
21290 root 1 44 0 33768K 3500K select 0 0:00 0.00% sshd
752 smmsp 1 20 0 10696K 2704K pause 0 0:00 0.00% sendmail
21294 root 1 20 0 10104K 2284K pause 1 0:00 0.00% csh
20790 mysql 1 8 0 7060K 1388K wait 1 0:00 0.00% sh
23574 root 1 44 0 12208K 5116K CPU2 2 0:00 0.00% top
800 root 1 5 0 5684K 996K ttyin 0 0:00 0.00% getty
799 root 1 5 0 5684K 996K ttyin 0 0:00 0.00% getty
801 root 1 5 0 5684K 996K ttyin 2 0:00 0.00% getty
798 root 1 5 0 5684K 996K ttyin 2 0:00 0.00% getty
803 root 1 5 0 5684K 996K ttyin 2 0:00 0.00% getty
806 root 1 5 0 5684K 944K ttydcd 2 0:00 0.00% getty
797 root 1 5 0 5684K 996K ttyin 1 0:00 0.00% getty
804 root 1 5 0 5684K 996K ttyin 2 0:00 0.00% getty
802 root 1 5 0 5684K 996K ttyin 3 0:00 0.00% getty
805 root 1 5 0 5684K 944K ttydcd 2 0:00 0.00% getty
449 root 1 44 0 2180K 472K select 0 0:00 0.00% devd
20895 mysql 1500 44 0 409M 250M ucond 1 0:00 0.00% mysqld

]]> By: Noah Everett http://www.mysqlperformanceblog.com/2009/03/19/mysql-random-freezes-could-be-the-query-cache/comment-page-1/#comment-523608 Noah Everett Sun, 29 Mar 2009 22:45:33 +0000 http://www.mysqlperformanceblog.com/?p=645#comment-523608 We are experiencing this freezing issue as well on FreeBSD 7.1 with MySQL 5.1.32. Its a pretty high traffic site, but I've never seen the DB server go above 20% CPU usage. Currently the master db is freezing up. Its replicating to 2 slave servers. When it freezes up it's top state shows "ucond" and cpu usage is 0% or very low so somehow its still doing some sort of work. Trying to connect to it via the command line mysql client returns error 34, Can't create a new thread Also issuing a /usr/local/etc/rc.d/mysql-server restart command just hangs as well, have to issue a kill -9 pid to make the server stop and restart. We were running FreeBSD 7.0 and MySql 5.0 and never saw this issue. Any help would be GREATLY appreciated, I honestly don't know where to start. Here is our my.cnf: skip-name-resolve max_connections = 2000 skip-innodb log-bin = /usr/mysql/logs/mysql-bin.log binlog-do-db=tp server-id=1 wait_timeout = 15 key_buffer = 16M key_buffer_size = 32M max_allowed_packet = 16M thread_stack = 128K thread_cache_size = 64 query_cache_limit = 0 #query_cache_size = 64M query_cache_size = 0 query_cache_type = 0 join_buffer_size = 512K log_slow_queries = /usr/mysql/logs/mysql-slow.log We are experiencing this freezing issue as well on FreeBSD 7.1 with MySQL 5.1.32. Its a pretty high traffic site, but I’ve never seen the DB server go above 20% CPU usage. Currently the master db is freezing up. Its replicating to 2 slave servers. When it freezes up it’s top state shows “ucond” and cpu usage is 0% or very low so somehow its still doing some sort of work. Trying to connect to it via the command line mysql client returns error 34, Can’t create a new thread

Also issuing a /usr/local/etc/rc.d/mysql-server restart command just hangs as well, have to issue a kill -9 pid to make the server stop and restart. We were running FreeBSD 7.0 and MySql 5.0 and never saw this issue. Any help would be GREATLY appreciated, I honestly don’t know where to start.

Here is our my.cnf:
skip-name-resolve
max_connections = 2000
skip-innodb
log-bin = /usr/mysql/logs/mysql-bin.log
binlog-do-db=tp
server-id=1

wait_timeout = 15
key_buffer = 16M
key_buffer_size = 32M
max_allowed_packet = 16M
thread_stack = 128K
thread_cache_size = 64
query_cache_limit = 0
#query_cache_size = 64M
query_cache_size = 0
query_cache_type = 0
join_buffer_size = 512K
log_slow_queries = /usr/mysql/logs/mysql-slow.log

]]> By: Salman Akram http://www.mysqlperformanceblog.com/2009/03/19/mysql-random-freezes-could-be-the-query-cache/comment-page-1/#comment-521112 Salman Akram Fri, 27 Mar 2009 18:46:50 +0000 http://www.mysqlperformanceblog.com/?p=645#comment-521112 Hi, I am working on a system which has currently around 20gb of data (increasing at the rate of ~200mb/day). We need to save complete documents in the system so basically one column has around 70-80% of the data. Our server is Quad Dual Core 4GB Ram, Server 2003 and using MySQL 5.0. Query cache size is 156M and limit is 8M. All tables except one are INNO DB and its buffer pool size is 1024M. The end user client module is read-only so lots and lots of repeated queries therefore I think query cache helps a lot but one of the problems I am facing is somewhat similar as discussed above. Sometimes the system halts for few mins. I fear that if I disable the query cache it will slow down my system which already is just OK. Apart from that I also want to handle the future size of the db which is increasing rapidly. I have got hold of High Performance MySQL but any hints where should I be looking at first? Any help will he highly appreciated. I can also give more details of my system in case if it will help. Thanks a lot! Hi,

I am working on a system which has currently around 20gb of data (increasing at the rate of ~200mb/day). We need to save complete documents in the system so basically one column has around 70-80% of the data. Our server is Quad Dual Core 4GB Ram, Server 2003 and using MySQL 5.0. Query cache size is 156M and limit is 8M. All tables except one are INNO DB and its buffer pool size is 1024M.

The end user client module is read-only so lots and lots of repeated queries therefore I think query cache helps a lot but one of the problems I am facing is somewhat similar as discussed above. Sometimes the system halts for few mins. I fear that if I disable the query cache it will slow down my system which already is just OK.

Apart from that I also want to handle the future size of the db which is increasing rapidly. I have got hold of High Performance MySQL but any hints where should I be looking at first?

Any help will he highly appreciated. I can also give more details of my system in case if it will help. Thanks a lot!

]]> By: Kristofer http://www.mysqlperformanceblog.com/2009/03/19/mysql-random-freezes-could-be-the-query-cache/comment-page-1/#comment-516145 Kristofer Mon, 23 Mar 2009 12:47:22 +0000 http://www.mysqlperformanceblog.com/?p=645#comment-516145 Mark: I don't think you're missing anything. I just meant that there are different mutexes and locks used in the query cache and not just one single mutex, and you seem to agree on that. I think the main focus should be on the memory allocator part since it (1) contains a linear search, (2) put too much work in getting the smallest memory block possible for the cache, (3) also has a problem with fragmentation and (4) causes increased code complexity where we could use standard components (maybe a google allocator?). Another thing which could easily be done is to start using a "lock-free" (== HW atomic compare and switch) hash to speed up searches. How much performance gain we get from this isn't trivial though given all the integration constrains. Saying that we should remove struct_guard_mutex is ok, but how do you want to have it done instead if we don't use that mutex to protect memory? Mark: I don’t think you’re missing anything. I just meant that there are different mutexes and locks used in the query cache and not just one single mutex, and you seem to agree on that. I think the main focus should be on the memory allocator part since it (1) contains a linear search, (2) put too much work in getting the smallest memory block possible for the cache, (3) also has a problem with fragmentation and (4) causes increased code complexity where we could use standard components (maybe a google allocator?). Another thing which could easily be done is to start using a “lock-free” (== HW atomic compare and switch) hash to speed up searches. How much performance gain we get from this isn’t trivial though given all the integration constrains. Saying that we should remove struct_guard_mutex is ok, but how do you want to have it done instead if we don’t use that mutex to protect memory?

]]> By: Brendan http://www.mysqlperformanceblog.com/2009/03/19/mysql-random-freezes-could-be-the-query-cache/comment-page-1/#comment-513836 Brendan Fri, 20 Mar 2009 23:49:08 +0000 http://www.mysqlperformanceblog.com/?p=645#comment-513836 Antony: What about keying off of *both* the actual query and a cryptographic digest of the tokens? I.e. attempt to hit the cache as the current implementation does. If that misses, go through the parsing stage & check again with the digest. Upside: same cost in terms of parsing, higher hit rate. Downside: storing 2x keys, performing 2x lookups for true misses. Antony:

What about keying off of *both* the actual query and a cryptographic digest of the tokens? I.e. attempt to hit the cache as the current implementation does. If that misses, go through the parsing stage & check again with the digest.

Upside: same cost in terms of parsing, higher hit rate.
Downside: storing 2x keys, performing 2x lookups for true misses.

]]> By: Mark Callaghan http://www.mysqlperformanceblog.com/2009/03/19/mysql-random-freezes-could-be-the-query-cache/comment-page-1/#comment-513759 Mark Callaghan Fri, 20 Mar 2009 23:12:26 +0000 http://www.mysqlperformanceblog.com/?p=645#comment-513759 @Kristofer -- What am I missing? structure_guard_mutex is locked in Query_cache::send_result_to_client() for any thread that queries the cache. It is also locked for much of the work done by Query_cache::store_query(). There are also read-write locks, but I have not spent the time to understand them. What release will contain the performance fixes? 6.X is a long way off. @Kristofer — What am I missing? structure_guard_mutex is locked in Query_cache::send_result_to_client() for any thread that queries the cache. It is also locked for much of the work done by Query_cache::store_query(). There are also read-write locks, but I have not spent the time to understand them.

What release will contain the performance fixes? 6.X is a long way off.

]]> By: Kristofer http://www.mysqlperformanceblog.com/2009/03/19/mysql-random-freezes-could-be-the-query-cache/comment-page-1/#comment-513674 Kristofer Fri, 20 Mar 2009 22:37:19 +0000 http://www.mysqlperformanceblog.com/?p=645#comment-513674 Mark: "One mutex protecting the entire cache" isn't really true. There are several mutexes involved in its design. One of the major issues is the custom memory allocator which is a great idea for small embedded systems and for the initial targets of the QC. Changing the memory allocation model to something more standard which would allow for concurrent access would improve the situation a lot and reduce complexity. After all it just a fancy container. Using a lock free hash is also a good idea. We have a re-engineering team working on redesigning the QC from scratch. Brainstorming, wish-lists and requirements are of course both fun and helpful. I'm sure they would appreciate it. Anthony: The spin lock approach was a really silly idea. I'm the guilty guy behind it. Sorry ;-) At the time I recall facing some very strange requirements, and that is my only defense. Not my finest moment probably, but thanks to the power of open source, what doesn't kill us sure makes us stronger. It is probably time to revisit that code with a new fix. (The spin lock approach isn't in 5.1+) Arjen: We always try to do our best. There is no conspiracy that I know of :) You're contribution was appreciated! In defense of the design of the QC I must say that it was initially created to solve some specific performance issues on a certain kind of hardware and it proved to be a good idea for many people at that time (and still can be if you understand what you do). Now as we move towards a new generation of hardware, things look different. Mark: “One mutex protecting the entire cache” isn’t really true. There are several mutexes involved in its design. One of the major issues is the custom memory allocator which is a great idea for small embedded systems and for the initial targets of the QC. Changing the memory allocation model to something more standard which would allow for concurrent access would improve the situation a lot and reduce complexity. After all it just a fancy container. Using a lock free hash is also a good idea. We have a re-engineering team working on redesigning the QC from scratch. Brainstorming, wish-lists and requirements are of course both fun and helpful. I’m sure they would appreciate it.

Anthony: The spin lock approach was a really silly idea. I’m the guilty guy behind it. Sorry ;-) At the time I recall facing some very strange requirements, and that is my only defense. Not my finest moment probably, but thanks to the power of open source, what doesn’t kill us sure makes us stronger. It is probably time to revisit that code with a new fix. (The spin lock approach isn’t in 5.1+)

Arjen: We always try to do our best. There is no conspiracy that I know of :) You’re contribution was appreciated!

In defense of the design of the QC I must say that it was initially created to solve some specific performance issues on a certain kind of hardware and it proved to be a good idea for many people at that time (and still can be if you understand what you do). Now as we move towards a new generation of hardware, things look different.

]]>