September 8, 2006

Why Index could refuse to work ?

Posted by peter |

Have you ever seen index which refused to be used even if there is every reason for it to work (from the glance view):

PLAIN TEXT
SQL:
  1. mysql> EXPLAIN SELECT * FROM article WHERE article_id=10;
  2. +----+-------------+---------+------+---------------+------+---------+------+-------+-------------+
  3. | id | select_type | TABLE   | type | possible_keys | KEY  | key_len | ref  | rows  | Extra       |
  4. +----+-------------+---------+------+---------------+------+---------+------+-------+-------------+
  5. |  1 | SIMPLE      | article | ALL  | PRIMARY       | NULL | NULL    | NULL | 93490 | USING WHERE |
  6. +----+-------------+---------+------+---------------+------+---------+------+-------+-------------+
  7. 1 row IN SET (0.00 sec)

Why on the earth index would not be used you would think, even if MySQL is mentioning it in "possible keys" ? Should you try to force it ?

PLAIN TEXT
SQL:
  1. mysql> EXPLAIN SELECT * FROM article force INDEX (PRIMARY) WHERE article_id=10;
  2. +----+-------------+---------+------+---------------+------+---------+------+-------+-------------+
  3. | id | select_type | TABLE   | type | possible_keys | KEY  | key_len | ref  | rows  | Extra       |
  4. +----+-------------+---------+------+---------------+------+---------+------+-------+-------------+
  5. |  1 | SIMPLE      | article | ALL  | PRIMARY       | NULL | NULL    | NULL | 93490 | USING WHERE |
  6. +----+-------------+---------+------+---------------+------+---------+------+-------+-------------+
  7. 1 row IN SET (0.00 sec)

No Luck. Even Force Index can't cure the problem. So what could it be ?

Lets take a look at article table:

PLAIN TEXT
SQL:
  1. CREATE TABLE `article` (
  2.   `article_id` varchar(20) NOT NULL,
  3.   `dummy` varchar(255) NOT NULL DEFAULT 'dummy',
  4.   PRIMARY KEY  (`article_id`)
  5. )

As you can see article_id is VARCHAR and this is the problem. Comparing String to Number is not going to use the index. Lets check if your guess is right:

PLAIN TEXT
SQL:
  1. mysql> EXPLAIN SELECT * FROM article WHERE article_id="10";
  2. +----+-------------+---------+------+---------------+---------+---------+-------+------+-------------+
  3. | id | select_type | TABLE   | type | possible_keys | KEY     | key_len | ref   | rows | Extra       |
  4. +----+-------------+---------+------+---------------+---------+---------+-------+------+-------------+
  5. |  1 | SIMPLE      | article | ref  | PRIMARY       | PRIMARY | 62      | const |    1 | USING WHERE |
  6. +----+-------------+---------+------+---------------+---------+---------+-------+------+-------------+
  7. 1 row IN SET (0.00 sec)

Looks much better does not it ?

So why you would define something as VARCHAR and when to refer to it as an INTEGER ? I see no good reason but It is quite frequently seen in applications. Might be designers just think - lets put it VARCHAR so it will fit strings if we need it to, but later decide to stick to numbers.

If you're storing INTEGER it is much better to define your columns as INT for many reasons, but if you decided to go with VARCHAR (ie you need leading zeroes to be preserved) you should refer to it as a sting in your application by using "".

You may ask why MySQL can't use index in this case, simply by converting number to the string and performing index lookup ?
This can't be done as it would result in wrong result for some queries. The thing is there are multiple strings possible for single integer value - for example for there is number 5 and strings "5", "05" "0005" "5.0" which all have same numeric value but different strings - this is why simple coversion to the string does not work.

Interesing enough it works other way around - you can refer to integer column as a string in most cases and MySQL will use the index, as for any string there is only one number which matches it. I guess this causes a lot of confusions - having seen it working in one direction people assume it also works in reverse one.

To add confusion MySQL mentions key as "possible keys" while really it has no way to use it for lookup (it can do index scan though). I guess "possible keys" are calculated before type matching is checked.

So be careful to use matching reference types in your applications :)

Posted by peter @ 9:40 am :: tips  
Print This Post Print This Post book mark Why Index could refuse to work ? in del.icio.usdel.icio.us :: submit Why Index could refuse to work ? to digg.comdigg  
Comment RSS :: Trackback URI  
Related posts: :Covering index and prefix indexes::How to check MySQL Config files::What is stored InnoDB buffer pool:
 

9 Comments »

  1. 1. Daniel Schneller

    Yes, this is really a great one. I have seen this in an application that performed searches on tables with millions of records and even when specifiying the exact primary key value, it took tens of seconds, not to mention joins.
    From my experience the “let’s take a varchar” approach is often taken by people who come from simpler systems, e. g. (in my example) Foxpro, where they stored nearly everything as character values.

    Comment :: September 8, 2006 @ 11:44 am

  2. 2. peter

    I did not mention it in the article but a lot of such problems also come from comparing incompatible columns, ie varchar to integer which may require closer look to figure out. I would really like MySQL EXPLAIN to issue warning in such case or something similar so it is easier to track it down.

    Yes this mistake is usually done by people having limited experience, at least with MySQL.

    Comment :: September 8, 2006 @ 12:25 pm

  3. 3. parvesh

    Sorry for commenting on an old post. But recently, I saw another interesting thing. You are right about indexes not being used in this case, but it works the other way round. If the column is an integer and the query is something like

    SELECT * FROM article WHERE article_id=”10″;

    My first intuition was that this will also fail, but MySQL is smart enough :)

    Comment :: July 3, 2007 @ 5:14 am

  4. 4. peter

    No this one is actually fine, because there is only one number for given string “10″ is 10 but it is not the case other way around 10 can correspond to “10″ “010″ “10.0″ and a lot of other strings.

    Comment :: July 3, 2007 @ 7:08 am

  5. 5. Jonathan

    Thank you, this is a great post. It just caught me in a system!

    Comment :: November 24, 2008 @ 11:49 pm

  6. 6. Suman

    What will happen if the table is:
    CREATE TABLE `article` (
    `article_id` int(5) NOT NULL,
    `dummy` varchar(255) NOT NULL DEFAULT ‘dummy’,
    PRIMARY KEY (`article_id`)
    )
    And query is:
    SELECT * FROM article force INDEX (PRIMARY) WHERE article_id=’10′; (single quotes)
    Will the index be used?

    Comment :: February 19, 2009 @ 6:47 am

  7. 7. Suman

    Sorry this has been answered above, I missed it.

    Comment :: February 19, 2009 @ 6:50 am

  8. 8. Peter Mouland

    Hi, any other reasons why FORCE INDEX is ignored? I have a table with 100 columns which about 25% are integers and most of those have indexes on them. A varchar column is being used as to left join to another table which also has an (btree) index on it, but for some reason it is not being used. EXPLAIN also doesn’t tell me about any possible keys. The table has about 100,000 rows, so not huge but be steadily growing. any ideas?? thanks,pete

    Comment :: June 15, 2009 @ 3:11 am

  9. 9. peter

    Pete,

    This may happen if MySQL can’t use this index for the query.
    FORCE INDEX only works if it is possible to use index but MySQL choses not to do that.

    Comment :: June 15, 2009 @ 8:55 pm

 

Subscribe without commenting

Trackbacks/Pingbacks