April 17, 2014

How to move the InnoDB log sequence number (LSN) forward

This post focuses on the problem of the InnoDB log sequence number being in the future. Preface: What is an InnoDB log sequence number? The Log sequence number (LSN) is an important database parameter used by InnoDB in many places. The most important use is for crash recovery and buffer pool purge control. Internally, the InnoDB […]

Recovery deleted ibdata1

Recently I had a case when a customer deleted the InnoDB main table space – ibdata1 – and redo logs – ib_logfile*. MySQL keeps InnoDB files open all the time. The following recovery technique is based on this fact and it allowed to salvage the database. Actually, the files were deleted long time ago – […]

Data Corruption, DRBD and story of bug

Working with customer, I faced pretty nasty bug, which is actually not rare situation , but in this particular there are some lessons I would like to share. The case is pretty much described in bug 55981, or in pastebin. Everything below is related to InnoDB-plugin/XtraDB, but not to regular InnoDB ( i.e in MySQL […]

Why you can’t rely on a replica for disaster recovery

A couple of weeks ago one of my colleagues and I worked on a data corruption case that reminded me that sometimes people make unsafe assumptions without knowing it. This one involved SAN snapshotting that was unsafe. In a nutshell, the client used SAN block-level replication to maintain a standby/failover MySQL system, and there was […]

Recovery beyond data restore

Quite frequently I see customers looking at recovery as on ability to restore data from backup which can be far from being enough to restore the whole system to operating state, especially for complex systems. Instead of looking just at data restore process you better look at the whole process which is required to bring […]

Recovering Innodb table Corruption

Assume you’re running MySQL with Innodb tables and you’ve got crappy hardware, driver bug, kernel bug, unlucky power failure or some rare MySQL bug and some pages in Innodb tablespace got corrupted. In such cases Innodb will typically print something like this: InnoDB: Database page corruption on disk or a failed InnoDB: file read of […]

Innodb Recovery Update – The tricks what failed.

As I wrote we had been recovering corrupted Innodb Tablespace and it is finally done now. As this was over than 1TB worth of data we really tried to avoid dumping the data and find some other way to recovery. Examining Innodb page content and crash information we figured out it should be page of […]

Innodb crash recovery update

I have not had a serious Innodb corruptions for a while, typically even if it happened it was some simple table related corruption which was easy to fix on table level. In couple of cases during last year when it was more than that we had backups and binary logs which means it was easier […]

Magic Innodb Recovery self healing

We have certain type of the table corrupting with Innodb, as it is limited to only one particular index on one particular table type it is likely to be Innodb bug but Heikki currently could not find what could be causing it. Happily as we have data stored in many tables of same format rather […]

Innodb Double Write

One of very interesting techniques Innodb uses is technique called “doublewrite” It means Innodb will write data twice when it performs table space writes – writes to log files are done only once. So why doublewrite is needed ? It is needed to archive data safety in case of partial page writes. Innodb does not […]