This is the good question. Actually a while ago MySQL would just start listening on the socket but connection will not succeed until recovery is completed. Right now it will not even start accepting connections until storage engines are initialized which is better.

Accepting them but refusing queries to say Innodb tables with “Recovery In Progress” or staling them on table lock could be an option but for most of the applications it would complicate things you have to think about and monitor - right now you have either MySQL is down or it is up and fully operational which is easy to check.

In your case I guess you need to check MySQL Error logs as well to see if you need to reboot the box. This requires some hacking for remote boxes but should not be that bad.

Your connection will fail, but you have no idea why. We have an embedded appliance, that tries to connect serveral times with a rudimentary backoff algorithm. If we can’t connect, we just shutdown the server, or in the extreme case reboot the box.

So we ran into a problem when scale testing that the recovery was taking quite a bit of time, as expected, but the box would get rebooted underneath it. So recovery would fail, and you go through this again the next time you start up the system / database.

So what is the proper way to do this without user intervetion? Is there a way I can see the recovery is going on, maybe with a different error code, so I should at least know to retry, and the database is not really in a hung state.

Indeed changing innodb_log_file_size is most common case of this problem, or related issues such as restoring from backup when log file size was different etc.

I think this is a bit artificial case. If Innodb crashes and it restarts it normally would self recover anyway. If it is bad database corruption so recovery does not work Innodb will in most cases assert and crash, ie due to page read failed.

Innodb fails to initialized and is disabled typically in case of configuration changes permission problems etc.

Now even imagine if you have Innodb as a master - for most people using basic monitoring people will not detect master is down if it is just Innodb which failed to initialize. And if you’re good with things and use custom monitoring and fall back scripts you can handle it by starting system with –skip-innodb to perform log shipping unless you use some other way for it.