Comments on: How to decrease InnoDB shutdown times http://www.mysqlperformanceblog.com/2009/04/15/how-to-decrease-innodb-shutdown-times/ Everything about MySQL Performance Sat, 21 Nov 2009 05:23:57 -0800 http://wordpress.org/?v=2.8.4 hourly 1 By: kerja keras http://www.mysqlperformanceblog.com/2009/04/15/how-to-decrease-innodb-shutdown-times/comment-page-1/#comment-666392 kerja keras Mon, 19 Oct 2009 03:39:12 +0000 http://www.mysqlperformanceblog.com/?p=660#comment-666392 great tips, its very helpful. thanks for share great tips, its very helpful. thanks for share

]]> By: Baron Schwartz http://www.mysqlperformanceblog.com/2009/04/15/how-to-decrease-innodb-shutdown-times/comment-page-1/#comment-561186 Baron Schwartz Thu, 14 May 2009 17:36:46 +0000 http://www.mysqlperformanceblog.com/?p=660#comment-561186 Kayra, it can become negative because the insert buffer survives after restart and merging can continue after the server is started again. Peter tells me he has seen servers take weeks to catch up in some cases! Kayra, it can become negative because the insert buffer survives after restart and merging can continue after the server is started again. Peter tells me he has seen servers take weeks to catch up in some cases!

]]> By: Kayra Otaner http://www.mysqlperformanceblog.com/2009/04/15/how-to-decrease-innodb-shutdown-times/comment-page-1/#comment-561178 Kayra Otaner Thu, 14 May 2009 16:16:50 +0000 http://www.mysqlperformanceblog.com/?p=660#comment-561178 Even after all the updating threads shut down, you'll see Innodb_buffer_pool_pages_dirty count approaching to 0 but still increasing sometimes. This is because of InnoDB merges, you can observe this using 'show engine innodb status\G'. Only when all the merges are completed pages_dirty count will be 0 and stay at zero. To calculate how many merges still need to be done : Under "INSERT BUFFER AND ADAPTIVE HASH INDEX" section difference between 'Inserts' and 'merged recs' will show how many to go. Example : ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 4185, free list len 3748, seg size 7934, 65171060 inserts, 65171060 merged recs, 10552858 merges Hash table size 7055701, used cells 32410, node heap has 33 buffer(s) 65171060 - 65171060 = 0 this means that Innodb_buffer_pool_pages_dirty should be 0 and stay at 0. Please note this calculation works most of the time, but sometimes 'inserts' - 'merged recs' value goes negative values. Even after all the updating threads shut down, you’ll see Innodb_buffer_pool_pages_dirty count approaching to 0 but still increasing sometimes. This is because of InnoDB merges, you can observe this using ’show engine innodb status\G’. Only when all the merges are completed pages_dirty count will be 0 and stay at zero. To calculate how many merges still need to be done :
Under “INSERT BUFFER AND ADAPTIVE HASH INDEX” section difference between ‘Inserts’ and ‘merged recs’ will show how many to go. Example :

————————————-
INSERT BUFFER AND ADAPTIVE HASH INDEX
————————————-
Ibuf: size 4185, free list len 3748, seg size 7934,
65171060 inserts, 65171060 merged recs, 10552858 merges
Hash table size 7055701, used cells 32410, node heap has 33 buffer(s)

65171060 – 65171060 = 0

this means that Innodb_buffer_pool_pages_dirty should be 0 and stay at 0. Please note this calculation works most of the time, but sometimes ‘inserts’ – ‘merged recs’ value goes negative values.

]]> By: Wagner Bianchi http://www.mysqlperformanceblog.com/2009/04/15/how-to-decrease-innodb-shutdown-times/comment-page-1/#comment-542369 Wagner Bianchi Sat, 18 Apr 2009 01:47:26 +0000 http://www.mysqlperformanceblog.com/?p=660#comment-542369 This is a very good idea. So nice. This is a very good idea.
So nice.

]]> By: Kevin Burton http://www.mysqlperformanceblog.com/2009/04/15/how-to-decrease-innodb-shutdown-times/comment-page-1/#comment-541960 Kevin Burton Fri, 17 Apr 2009 16:51:30 +0000 http://www.mysqlperformanceblog.com/?p=660#comment-541960 Another note... we have three replicas of everything. When we restart an InnoDB box it is taken out of production and does not serve requests. Then we flush the innodb buffer pools, then restart it. The script I linked to above is just added as a one liner to our /etc/init.d/mysql script.... if InnoDB isn't running on this box it isn't run. We have about 1/2 of our cluster on MyISAM without InnoDB enabled so this would break otherwise. Another note… we have three replicas of everything.

When we restart an InnoDB box it is taken out of production and does not serve requests.

Then we flush the innodb buffer pools, then restart it.

The script I linked to above is just added as a one liner to our /etc/init.d/mysql script…. if InnoDB isn’t running on this box it isn’t run.

We have about 1/2 of our cluster on MyISAM without InnoDB enabled so this would break otherwise.

]]> By: Kevin Burton http://www.mysqlperformanceblog.com/2009/04/15/how-to-decrease-innodb-shutdown-times/comment-page-1/#comment-541959 Kevin Burton Fri, 17 Apr 2009 16:49:03 +0000 http://www.mysqlperformanceblog.com/?p=660#comment-541959 Hey Baron.... Here's the script we're using: http://pastebin.com/f5ec2f70d If we were to write it again it would almost certainly be written in python. Kevin Hey Baron….

Here’s the script we’re using:

http://pastebin.com/f5ec2f70d

If we were to write it again it would almost certainly be written in python.

Kevin

]]> By: Coolyou http://www.mysqlperformanceblog.com/2009/04/15/how-to-decrease-innodb-shutdown-times/comment-page-1/#comment-541528 Coolyou Fri, 17 Apr 2009 03:29:21 +0000 http://www.mysqlperformanceblog.com/?p=660#comment-541528 Kevin, we were about to write one of these scripts ourselves until we realized that we may have an issue with figuring out how far we are with the flushing. Could you please provide us all with the script? It may prove very useful :) Pat, the whole idea of this is to keep the application online for a longer period of time than having MySQL refuse all connections for that 15 minutes. However I have noticed that MySQL performance decreases dramatically when innodb_max_dirty_pages_pct = 0 is in effect. That's not unexpected considering that the dirty pages are immediately flushed to disk. But those with a busy site may want to consider this limitation because this option may even increase the time required to shut down because of the increased load from the application itself and the increased time it takes for the application to respond. Kevin, we were about to write one of these scripts ourselves until we realized that we may have an issue with figuring out how far we are with the flushing. Could you please provide us all with the script? It may prove very useful :)

Pat, the whole idea of this is to keep the application online for a longer period of time than having MySQL refuse all connections for that 15 minutes.

However I have noticed that MySQL performance decreases dramatically when innodb_max_dirty_pages_pct = 0 is in effect. That’s not unexpected considering that the dirty pages are immediately flushed to disk. But those with a busy site may want to consider this limitation because this option may even increase the time required to shut down because of the increased load from the application itself and the increased time it takes for the application to respond.

]]> By: Baron Schwartz http://www.mysqlperformanceblog.com/2009/04/15/how-to-decrease-innodb-shutdown-times/comment-page-1/#comment-541452 Baron Schwartz Fri, 17 Apr 2009 02:39:16 +0000 http://www.mysqlperformanceblog.com/?p=660#comment-541452 Pat, If your server takes 15 minutes to shut down, you can keep the application online and responding for some portion of that time, while dirty pages are flushed from the buffer pool. The alternative is to take the app offline for the whole 15 minutes. The point is to reduce the amount of time that the server is *actually* shutting down and restarting, by *preparing* it for shutdown. Kevin, can you post your shutdown script? Maybe put it on your blog and link to it here? Maybe contribute it to your favorite distro? It sounds useful. Pat,

If your server takes 15 minutes to shut down, you can keep the application online and responding for some portion of that time, while dirty pages are flushed from the buffer pool. The alternative is to take the app offline for the whole 15 minutes. The point is to reduce the amount of time that the server is *actually* shutting down and restarting, by *preparing* it for shutdown.

Kevin, can you post your shutdown script? Maybe put it on your blog and link to it here? Maybe contribute it to your favorite distro? It sounds useful.

]]> By: Kevin Burton http://www.mysqlperformanceblog.com/2009/04/15/how-to-decrease-innodb-shutdown-times/comment-page-1/#comment-541283 Kevin Burton Fri, 17 Apr 2009 00:14:43 +0000 http://www.mysqlperformanceblog.com/?p=660#comment-541283 pat, If you're new to MySQL and your DB is taking 12 minutes to shutdown, and the port is closed, one might assume that the DB is broken and kill -9 it.. Our script actually prints that it's flushing the buffer pool and prints percentage until complete numbers. pat,

If you’re new to MySQL and your DB is taking 12 minutes to shutdown, and the port is closed, one might assume that the DB is broken and kill -9 it..

Our script actually prints that it’s flushing the buffer pool and prints percentage until complete numbers.

]]> By: pat http://www.mysqlperformanceblog.com/2009/04/15/how-to-decrease-innodb-shutdown-times/comment-page-1/#comment-541253 pat Thu, 16 Apr 2009 23:47:50 +0000 http://www.mysqlperformanceblog.com/?p=660#comment-541253 Is this really shutting down my server any faster in wall clock time? Without setting the max_dirty_pages flag to zero, say it takes me 15 minutes to shut down. If I do set it, doesn't it just take me 14 minutes for dirty pages to drop to zero and 1 minute for the shutdown? Seems like it might make the shutdown *command* take less time, but the actual elapsed time from the moment you decide to shut down the server to the time mysqld terminates isn't decreasing is it? Is this really shutting down my server any faster in wall clock time?

Without setting the max_dirty_pages flag to zero, say it takes me 15 minutes to shut down.

If I do set it, doesn’t it just take me 14 minutes for dirty pages to drop to zero and 1 minute for the shutdown?

Seems like it might make the shutdown *command* take less time, but the actual elapsed time from the moment you decide to shut down the server to the time mysqld terminates isn’t decreasing is it?

]]>