Comments on: Fixing column encoding mess in MySQL

By: codeAPE » Blog Archive » Fixing UTF8 data that was inserted using the latin1 encoding

Wed, 14 Jan 2009 03:49:43 +0000

[...] http://www.mysqlperformanceblog.com/2007/12/18/fixing-column-encoding-mess-in-mysql/ Posted in Computers | [...]

By: brigada

brigada — Sun, 19 Oct 2008 21:20:18 +0000

phpmyadmin makes this very easy to change – Open your database in phpmyadmin, select a table on the left, select the field(s) you need to change, click the With All: CHANGE button and change the collation. Depending on your amount of tables (i had 40) this takes about 10 min. max, and its error-free.

You can test it out on any old database you have if you wish before doing it ”live”

By: tamcy

tamcy — Tue, 02 Sep 2008 06:43:26 +0000

Sorry – answer to myself, but just realized the way to fix it and would like to share with the others.

As I said, the column expects utf8 data, and the data is correct when “set names latin1″ is issued.
This means I can get the correct (utf8) data when MySQL performs a “utf8 -> latin1″ conversion.
In order to fix the encoding I need to treat this “latin1″ data as binary, then convert it to utf8.
And so here is the statement I need:

UPDATE table SET col = convert(CONVERT(CONVERT(col USING latin1) USING binary) using utf8);

By: tamcy

tamcy — Tue, 02 Sep 2008 06:35:21 +0000

Perhaps a more complicated issue:

The table is created with charset utf8 – correct.
A data is exported with charset utf8 – also correct.
But thing goes wrong when trying to import the data into mysql with “\.” syntax. The user forgot to run “SET NAMES utf8″, and MySQL sees all data as latin1.

Now, the script can use “set names latin1″ to get back utf8 data.
However, “CONVERT(CONVERT(offending_col USING binary) USING utf8)” doesn’t work – the same wrong garbage character is shown.

By: andreea

andreea — Sun, 24 Aug 2008 13:16:01 +0000

salut

By: shodan

shodan — Fri, 11 Jul 2008 13:08:58 +0000

Jeroen,

this means that the data you’re marking as UTF-8 is *not* really in UTF-8 encoding, and then UTF-8 decoder chokes.

Eg. if the binary data is actually is in Latin-1 encoding but you’re tricking MySQL to think it’s in UTF-8, the decoder will choke at the very first non-ASCII7 symbol.

By: Jeroen Sen

Jeroen Sen — Fri, 11 Jul 2008 12:14:32 +0000

I’ve used your conversion a bit for converting a database characterset latin_1 to an utf_8 characterset. (Just like SCC above.) But I have a problem with some special characters like ‘ë’ and ‘ä’ for example. When these characters are converted to their binary counterpart they are present in the database. But when I convert the database into utf_8 all contents of the fields containing these symbols are droppen starting form the symbol. Does anyone have a suggestion?

Thnx in advance!

By: Jerk

Jerk — Sat, 23 Feb 2008 16:53:22 +0000

@Sven Neuhaus: That worked perfectly for Latin1 encoded chars in a UTF8 col. THANX!. That saved our lives.

By: SCC

SCC — Thu, 14 Feb 2008 06:17:15 +0000

I wish there was a way to change an entire table at once instead of going column by column. I need to convert from latin1_swedish_ci to UTF8 for dozens of columns.

By: shodan

shodan — Fri, 01 Feb 2008 00:03:49 +0000

Mohnkhan,
English is encoded exactly the same in both Latin-1 and UTF-8 so I’d simply go with everything in UTF-8 for unification.