Comments on: How to estimate time it takes Innodb to Recover ?

By: Ash

Ash — Thu, 26 Jun 2008 04:02:22 +0000

Hi all,

PROBLEM: mysqld frequently restarts w/o apparent or visible reasons yet:
Would this be related to long INNODB recovery time, to the point it will do self service-restart?

In hopes to clear confusions with the problem Im getting w/ our servers: Here is a snippet from our error.log:

************
080625 23:53:28 [Note] /usr/sbin/mysqld: ready for connections.
Version: ‘5.0.51a-community-log’ socket: ‘/var/lib/mysql/mysql.sock’ port: 3306 MySQL Community Edition (GPL)
080625 23:53:28 [Note] Slave SQL thread initialized, starting replication in log ‘bin.000025′ at position 12410321, relay log ‘/var/log/mysql/relay.000086′ position: 4837623
080625 23:53:28 [Note] Slave I/O thread: connected to master ‘repl@10.16.82.132:3306′, replication started in log ‘bin.000025′ at position 12410321

Number of processes running now: 0
080626 00:06:16 mysqld restarted
InnoDB: Log scan progressed past the checkpoint lsn 16 3276587130
080626 0:06:16 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files…
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer…
InnoDB: Doing recovery: scanned up to log sequence number 16 3276588706
080626 0:06:19 InnoDB: Starting an apply batch of log records to the database…
InnoDB: Progress in percents: 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 12501110, file name bin.000025
InnoDB: Last MySQL binlog file position 0 90914, file name /var/log/mysql/bin.000027
080626 0:06:20 InnoDB: Started; log sequence number 16 3276588706
080626 0:06:20 [Note] Recovering after a crash using /var/log/mysql/bin
080626 0:06:20 [Note] Starting crash recovery…
080626 0:06:20 [Note] Crash recovery finished.
080626 0:06:20 [Note] /usr/sbin/mysqld: ready for connections.
Version: ‘5.0.51a-community-log’ socket: ‘/var/lib/mysql/mysql.sock’ port: 3306 MySQL Community Edition (GPL)
080626 0:06:20 [Note] Slave SQL thread initialized, starting replication in log ‘bin.000025′ at position 12501137, relay log ‘/var/log/mysql/relay.000088′ position: 85644
080626 0:06:20 [Note] Slave I/O thread: connected to master ‘repl@10.16.82.132:3306′, replication started in log ‘bin.000025′ at position 12501137

**************

System background:
——————-
No of servers in replication chain: 20
-> 1a 2a 3a 4a 5a 6a 7a 8a 9a 10a ->
-> 1b 2b 3b 4b 5b 6b 7b 8b 9b 10b ->
where 10b is master of 1a

*** All db in replicated mode are using INNODB

OS (all): Linux 2.6.9-42.0.8.ELsmp #1 SMP Tue Jan 23 12:49:51 EST 2007 x86_64 x86_64 x86_64 GNU/Linux

Also currently: these config (my.cnf) were all disabled from 20 servers:
# Uncomment the following if you are using InnoDB tables
#innodb_data_home_dir = /var/lib/mysql/
#innodb_data_file_path = ibdata1:2000M;ibdata2:10M:autoextend
#innodb_log_group_home_dir = /var/lib/mysql/
#innodb_log_arch_dir = /var/lib/mysql/
# You can set .._buffer_pool_size up to 50 – 80 %
# of RAM but beware of setting memory usage too high
#innodb_buffer_pool_size = 384M
#innodb_additional_mem_pool_size = 20M
# Set .._log_file_size to 25 % of buffer pool size
#innodb_log_file_size = 100M
#innodb_log_buffer_size = 8M
#innodb_flush_log_at_trx_commit = 1
#innodb_lock_wait_timeout = 50

Thanks in adv,
Ash

By: MySQL Replication vs DRBD Battles | MySQL Performance Blog

MySQL Replication vs DRBD Battles | MySQL Performance Blog — Tue, 29 Apr 2008 03:51:02 +0000

[...] I would also say this implies hidden danger – the time it will take your database to do transactional log recovery is invisible until you get a crash, meaning if your production database size growths load changes or you happen to have failure during activity of certain kind you might have it taking much longer when expected. recovery time depends on a lot of variables. [...]

By: peter

peter — Fri, 07 Mar 2008 23:42:46 +0000

Well REDO phase of recovery is completed before MySQL starts to accept connections.
Undo phase is in background and is not really visible – you can find it in the error log. Also in SHOW INNODB STATUS you should see undo happening.

By: dac

dac — Fri, 07 Mar 2008 19:05:37 +0000

One issue I have which I have not found an answer to is this:

How do I know when innodb crash recovery has completed from a programmatic stand-point?

Typically on a system start up, mysqld is automatically started, then the application(s)
are started. However, if the database is doing recovery, the applications may fail to
connect (say via JDBC) for quite a while or may not start up at all.

How can one check if the database is in recovery mode before proceeding?

By: What is the longest part of Innodb Recovery Process ? | MySQL Performance Blog

Thu, 20 Dec 2007 10:25:42 +0000

[...] In MySQL 4.1 and above the longest part of recovery after crash for Innodb tables could be UNDO stage – it was happening in foreground and was basically unbound – if you have large enough transaction which needed to be undone this could take long hours. REDO stage on other hand always could be regulated by size of your Innodb log files so you could have it as large as you like. Read more about it here. [...]

By: MySQL Performance Blog » Innodb Recovery - Is large buffer pool always better ?

Tue, 17 Jul 2007 15:46:38 +0000

[...] positive, Innodb with large buffer pool to performing better. Including Recovery of course. I even blogged about [...]

By: Innodb Recovery « François Schiettecatte’s Blog

Innodb Recovery « François Schiettecatte’s Blog — Wed, 09 May 2007 21:34:07 +0000

[...] From experience, I can corroborate that Innodb recovery can take what seems to be a fairly random amount of time to [...]