July 22, 2014

Why Index could refuse to work ?

Have you ever seen index which refused to be used even if there is every reason for it to work (from the glance view):

Why on the earth index would not be used you would think, even if MySQL is mentioning it in “possible keys” ? Should you try to force it ?

No Luck. Even Force Index can’t cure the problem. So what could it be ?

Lets take a look at article table:

As you can see article_id is VARCHAR and this is the problem. Comparing String to Number is not going to use the index. Lets check if your guess is right:

Looks much better does not it ?

So why you would define something as VARCHAR and when to refer to it as an INTEGER ? I see no good reason but It is quite frequently seen in applications. Might be designers just think – lets put it VARCHAR so it will fit strings if we need it to, but later decide to stick to numbers.

If you’re storing INTEGER it is much better to define your columns as INT for many reasons, but if you decided to go with VARCHAR (ie you need leading zeroes to be preserved) you should refer to it as a sting in your application by using “”.

You may ask why MySQL can’t use index in this case, simply by converting number to the string and performing index lookup ?
This can’t be done as it would result in wrong result for some queries. The thing is there are multiple strings possible for single integer value – for example for there is number 5 and strings “5″, “05″ “0005″ “5.0″ which all have same numeric value but different strings – this is why simple coversion to the string does not work.

Interesing enough it works other way around – you can refer to integer column as a string in most cases and MySQL will use the index, as for any string there is only one number which matches it. I guess this causes a lot of confusions – having seen it working in one direction people assume it also works in reverse one.

To add confusion MySQL mentions key as “possible keys” while really it has no way to use it for lookup (it can do index scan though). I guess “possible keys” are calculated before type matching is checked.

So be careful to use matching reference types in your applications :)

About Peter Zaitsev

Peter managed the High Performance Group within MySQL until 2006, when he founded Percona. Peter has a Master's Degree in Computer Science and is an expert in database kernels, computer hardware, and application scaling.

Comments

  1. Yes, this is really a great one. I have seen this in an application that performed searches on tables with millions of records and even when specifiying the exact primary key value, it took tens of seconds, not to mention joins.
    From my experience the “let’s take a varchar” approach is often taken by people who come from simpler systems, e. g. (in my example) Foxpro, where they stored nearly everything as character values.

  2. peter says:

    I did not mention it in the article but a lot of such problems also come from comparing incompatible columns, ie varchar to integer which may require closer look to figure out. I would really like MySQL EXPLAIN to issue warning in such case or something similar so it is easier to track it down.

    Yes this mistake is usually done by people having limited experience, at least with MySQL.

  3. parvesh says:

    Sorry for commenting on an old post. But recently, I saw another interesting thing. You are right about indexes not being used in this case, but it works the other way round. If the column is an integer and the query is something like

    SELECT * FROM article WHERE article_id=”10″;

    My first intuition was that this will also fail, but MySQL is smart enough :)

  4. peter says:

    No this one is actually fine, because there is only one number for given string “10″ is 10 but it is not the case other way around 10 can correspond to “10″ “010″ “10.0″ and a lot of other strings.

  5. Jonathan says:

    Thank you, this is a great post. It just caught me in a system!

  6. Suman says:

    What will happen if the table is:
    CREATE TABLE article (
    article_id int(5) NOT NULL,
    dummy varchar(255) NOT NULL DEFAULT ‘dummy’,
    PRIMARY KEY (article_id)
    )
    And query is:
    SELECT * FROM article force INDEX (PRIMARY) WHERE article_id=’10′; (single quotes)
    Will the index be used?

  7. Suman says:

    Sorry this has been answered above, I missed it.

  8. Peter Mouland says:

    Hi, any other reasons why FORCE INDEX is ignored? I have a table with 100 columns which about 25% are integers and most of those have indexes on them. A varchar column is being used as to left join to another table which also has an (btree) index on it, but for some reason it is not being used. EXPLAIN also doesn’t tell me about any possible keys. The table has about 100,000 rows, so not huge but be steadily growing. any ideas?? thanks,pete

  9. peter says:

    Pete,

    This may happen if MySQL can’t use this index for the query.
    FORCE INDEX only works if it is possible to use index but MySQL choses not to do that.

  10. mylesmg says:

    One reason not in this post, but in another post is that if you are joining tables, the join may not use an index if the tables are of different types (e.g. joining an InnoDB table to a Memory table). Simply changing the memory to innodb or vice versa will eliminate the issue.

  11. Mark Cotner says:

    This came up in a google search for a problem I was having where even if I forced I couldn’t get it to use an index. I quietly read the ENTIRE article, assuming this couldn’t possibly be it . . . surely we’re smarter than that.

    Went back and checked anyway, sure enough . . . varchar for a universal_id field. I guess some of them must have characters in them(we don’t assign them). Added quotes to the where clause and sped it up 300x.

    Thanks for posting this. It may seem obvious, but I surely didn’t catch it.

  12. Pascal says:

    The same problem i have, at the only difference, that my column is Already an INT….
    What should i do?

    Thanks!
    Pascal

  13. Yakovenko Stepan says:

    Would you be so kind to take a look here: http://stackoverflow.com/questions/13691422/mysql-with-huge-tables-2-queries-faster-then-1-index-not-used ? Can this happen to bigint field?

  14. sivamurugan says:

    Thanks a lot Peter Zaitsev.. It saves me in getting late of 6 million records…

Speak Your Mind

*