Comments on: Debugging problems with row based replication

By: Alexandr Yatsuk Leonidovich

Alexandr Yatsuk Leonidovich — Wed, 08 Sep 2010 00:26:00 +0000

Hi All.
I using MIXED binlog format. And having problem with simple delete query.
DELETE FROM temp.tenp WHERE id = 10;
This query must delete 3000000 rows from my innodb table.
But slave came to out of sync. (like in comment#2 by Justin Swanhart were hi used STATEMENT binlog format)

But if I’ve change binlog format to ROW based, I’ve got error like in comment#8 by Kim Carlsen …..

What I must to do ? Some configuratuin tuning?
do not delete a lot of rows in one query ?
How much rows I can delete per query if I using MIXED binlog format ?

By: Justin Swanhart

Justin Swanhart — Wed, 21 Jul 2010 19:41:38 +0000

Hi Kim,

I wanted to let you know that the next release of mysqlbinlog should include a –max-allowed-packet option instead of hard coding it to 1G. This should resolve your problem with mysqldump.

By: Kim Carlsen

Kim Carlsen — Wed, 12 May 2010 08:23:24 +0000

One problem I have experienced with ROW based binary logging, is that mysql can make log entries that get too huge to be parsed through mysql client.

An example is, if you delete all rows in a huge table
DELETE * FROM huge_table;

This will be formulated as one BINLOG ” entry in the binary log, with a delete statement for each row in the table.
As a result you will get the error ‘ERROR 1153 (08S01) at line 1074175: Got a packet bigger than ‘max_allowed_packet’ bytes’ when trying to apply
this log to a database (mysqlbinlog bin-log.000001 | mysql -uroot)

You can increase the max_allowed_packet to a maximum of 1GB, but you will still be able to get larger entries in the log.

By: peter

peter — Sat, 08 May 2010 00:17:14 +0000

Justin,

On Compatibility – I think this is why there is MySQL and Drizzle. MySQL has a lot of weight with bad choices over many years which can’t be easily broken because this unsettles a lot of customers who relay on these behaviors.

Also people do not often know they relay on any side effects. They just upgrade Master and Slave to MySQL 5.1 and just expect things work as before. If it breaks this is serious issue.
I think moving back to STATEMENT by default in MySQL 5.0 was a good choice. Breaking existing applications is a lot more serious issue than improving things (which row level replication does)

By: peter

peter — Sat, 08 May 2010 00:13:42 +0000

Justin,

With Statement Level replication MySQL 5.1 to 5.0 works for some kinds of applications. It is not guaranteed to work in all cases of course but it does for many simple MySQL use cases in practice.
This is in fact important case because it allows to get extra assurance for MySQL 5.0 -> 5.1 migrations – you can keep MySQL 5.0 slave for some time after promoting Master to MySQL 5.1 to have something to roll back to.

By: Justin Swanhart

Justin Swanhart — Fri, 07 May 2010 16:11:00 +0000

And I don’t buy the argument in: http://bugs.mysql.com/bug.php?id=39812

The people who were relying on statement side effects should manually use statement, or fix their application to set binlog_format=STATEMENT before statements where the stupid statement side-effects are desired.

By: Justin Swanhart

Justin Swanhart — Fri, 07 May 2010 16:01:07 +0000

Also, how is statement “more compatible”? You can’t safely replicate from a 5.1 to a 5.0 host, which is the only place the “compatibility” issue seems to make sense. A 5.1 slave can consume both Query and RBR events, so how does using statement improve anything? Triggers, cascading foreign keys and other indirect data changes are much more safely captured using RBR than statement logs. Using non-deterministic functions like UUID or RAND() are much safer with RBR.

By: Justin Swanhart

Justin Swanhart — Fri, 07 May 2010 15:53:57 +0000

And I’ve run into this problem when SQL_LOG_BIN=0 was set on the master and changes were made. Since deletions (or insertions) happened on the master that were not on the slave the slave was out of sync. By diagnosing the problem using this technique, I was able to track down how the rows went ‘missing’ on the slave.

My simple example is just that – a simple example. Reality is complex and there are very many reasons a slave can get out of sync. When it happens you need to know how to discover what isn’t in sync (which is not obvious from the error message) so that you can find the underlying cause and prevent it from happening again.

Further, it is possible to change from MIXED to ROW to STATEMENT in an individual session, thus a slave could get RBR updates even when the master is in STATEMENT mode.

By: Justin Swanhart

Justin Swanhart — Fri, 07 May 2010 15:43:21 +0000

The ‘human mistake’ is just one reason a slave can get out of sync…

Try this on your statement based master:
DELETE FROM some_table ORDER BY RAND() LIMIT 1000;

Your slave is now out of sync. Oops..

By: Giuseppe Maxia

Giuseppe Maxia — Fri, 07 May 2010 09:47:32 +0000

MySQL 5.1 default binlog_format in GA has always been STATEMENT. The main reason for this choice is that we wanted to avoid compatibility issues.
Notice that 5.1 has been GA since version 5.1.30.
I must say that your example is quite unsettling, as it is based on a human mistake that puts the slave in a non recoverable state.
See also this post, where the enhancement of mysqlbinlog was announced.
http://datacharmer.blogspot.com/2008/09/dsclosing-binlog-entries-with-row-based.html