Comments on: Converting Character Sets

By: Chris Huntley

Chris Huntley — Tue, 25 May 2010 16:52:56 +0000

Hi Ryan,
My blog is on a WordPress platform. Do you know if I’ll have to make the upgrade to utf8? Thanks.

By: bfarber

bfarber — Wed, 30 Dec 2009 23:14:25 +0000

While the general information in the article is useful, there’s 2 issues I’m spotting (since I recently undertook the same task).

1) Many databases store any charset text into the latin1 column, part of legacy MySQL 4 using latin1 as the default. While the text stores and can be displayed just fine from a front end application querying it, if you just change the character set I would assume MySQL is going to literally try to convert from latin1 to utf8, and in this case since the text isn’t latin1 it gets mangled.

2) I’ve seen many applications that store serialized data in the database (from PHP serialize() function). This method would break that serialized string, making it impossible to deserialize it later.

Just some things to point out.

By: nadavkav

nadavkav — Fri, 03 Apr 2009 17:32:54 +0000

Thank you ! for this valuable post.
But it was a little to late for me, after trying one of those suggestions on the internet
similar to the #2 method category for converting text to binary and back to text again.
We use a Mahara (ePortfolio framework) that was installed with the wrong charset
and got filled with data very quickly, before i had that chance to notice this issue.
eventually (after #2 method did not work properly) i had to manually change the relevant
DB fields
A good ending to this story is that everything works fine, now

By: Shlomi Noach

Shlomi Noach — Wed, 25 Mar 2009 17:26:37 +0000

@Gil,

good question. Will try to set up the configuration and test.

By: Gil

Gil — Wed, 25 Mar 2009 16:53:55 +0000

Thanks @Schlomi

My question was more about what happens if you have latin1 on the master and utf8 on the slave. Same problem?

By: Shlomi Noach

Shlomi Noach — Wed, 25 Mar 2009 16:51:18 +0000

@Gil
If you have a utf8 column on a master, and a latin1 column on a slave, and try to replicate, you may get corrupted data if you insert true utf8 text into the master. The slave will not be able to store utf8 characters and will truncate them.

By: Gil

Gil — Wed, 25 Mar 2009 16:08:38 +0000

In a multi-master replication setup, if I broke replication and ran this script on the passive master, then started replication from active->passive, would the queries fail or do other unwanted things? Are the binary logs charset-sensitive during replication?

By: Shlomi Noach

Shlomi Noach — Mon, 23 Mar 2009 15:36:35 +0000

@Ryan,

Cheers. There’s room for more than one utility per task!

Shlomi

By: Ryan Lowe

Ryan Lowe — Mon, 23 Mar 2009 14:30:34 +0000

@shlomi Had I seen that script sooner, I may have used it! convert_charset has –columns parameter, so it is possible to do one-and-and-only-one column at a time.

@jakub & @pavel This script assumes latin1 data is stored in latin1 columns:)

By: Pavel

Pavel — Mon, 23 Mar 2009 10:54:54 +0000

What happens if a column has mixed latin1 and utf8 at the same time (left from MySQL 4)?
Does MySQL detect that right?