select * from products where price > 10 and price 5 and weight <70 and color = 1 … order by created_on desc limit 10;

Note that all the conditions are dynamically generated and some of them may or may not be there, e.g., some people filter by price, some by color, some by unitprice, … numerous possible combinations. How would you index this type of tables? Appreciate it!

create table orders(
order_id int unsigned not null auto_increment primary key,
item_type_id tinyint unsigned not null,
locale_id smallint unsigned not null,
currency_id tinyint unsigned not null,
operator_id smallint unsigned not null,
order_amount decimal(10, 4) not null,
order_date datetime not null,
last_updated timestamp not null default current_timestamp on update current_timestamp,
foreign key(item_type_id) references item_types(item_type_id),
foreign key(locale_id) references locales(locale_id),
foreign key(currency_id) references currencies(currency_id),
foreign key(operator_id) references operators(operator_id)
);

Lets say you have a clue and design your OLTP database using at least 3rd normal form and integrity constraints. You realize that ISAM was perhaps a reasonable choice in 1965 for a production database involving inserts, updates and deletes but not so much in 2010. MySQL surprisingly supports declarative integrity constraints. But wait being MySQL there must be something about it that’s bizarre and going to bite you somehow. Well it also requires that an index be created on each referencing column. At first this may seem to be tolerable, especially for cases where you’re requiring that deletes / updates cascade from the parent to child tables – naturally an index on the referencing columns in the child tables would help there – well maybe. Often though, the database is designed such that this is not the case. Updates and deletes of primary key values are instead restricted – that is not permitted. So the indexes on the referencing columns are quite useless. Actually they’re more than useless they’re detrimental. Not only do these indexes slow down insert operations, the MySQL optimizer sometimes appears to become rather overwhelmed by the number of indexes and tables to choose from for moderately complex queries and selects an extraordinarily inefficient plan resulting in horrendous performance by using one of these foreign key index with low cardinality and somewhat even distribution first instead of a primary or unique key even if such a key exists and would result in 1 row matching. So to circumvent that you need to help it out with optimizer hints like straight_join to have it look at tables in the right order and come up with a sane choice (this is using version 5.0.45 btw). This great fun when you’re using an ORM like SQL Alchemy and want to avoid writing sql directly in application code or introducing things that tie you to a particular dbms.

So as this thread started out by alluding to lots of single column indexes generally aren’t a good idea, well MySQL forces this upon you if you actually implement a proper database design and use declarative syntax to do so.

Here’s an example table

create table orders(
order_id int unsigned not null auto_increment primary key,
item_type_id tinyint unsigned not null,
locale_id smallint unsigned not null,
currency_id tinyint unsigned not null,
operator_id smallint unsigned not null,
order_amount decimal(10, 4) not null,
order_date datetime not null,
last_updated timestamp not null default current_timestamp on update current_timestamp,
foreign key(item_type_id) references item_types(item_type_id),
foreign key(locale_id) references items(locale_id),
foreign key(currency_id) references items(currency_id),
foreign key(operator_id) references items(operator_id)
);

create index order_date_idx on orders(order_date);

Okay so we know that we’ll never delete or change item_type ids, locale ids, currency ids or operator ids. And even if we did it would be a big deal – requiring system down time etc. New ones may be added but that’s it. Lets say there are about 10 million rows in this orders table. Of the referenced columns there are 10 types of items, 5 locales they could be sold in presently, 3 currencies and 20 operators.

MySQL will create the following indexes:
unique index on order_id
index on item_type_id
index on locale_id
index on currency_id
index on operator_id

You have no control over this at all – aside from not using declarative integrity constraints. You could resort to the 1980′s early 90′s way of writing triggers to implement referential integrity but I hear triggers are also pretty slow themselves in MySQL too and writing a few hundred of those isn’t going to be to enjoyable at all.

Additionally I need an index on order_date to help with finding orders for a given date.

8 columns 6 indexes on single columns. This is silly. You can imagine a table with more columns and FKs and more indexes. Sure you can vertically partition the data but you’ll end up with the same indexes regardless.

The foreign key indexes are all low cardinality with roughly even distributions. Nearly always a poor choice to use for any query. This is why most if not all widely used databases do not require indexes on FKs. The database designer should be able to choose what to do. Having them by default might be ok but they should not be required.

To solve sorting issue on big sets, people can use pre-explain. Then decide either use hint to force ordered access or access based on conditions. To avoid explains local in memory statistics can be used to calculate conditions selectivity.

Anyway, question was about set of indexes with compound vs. single context. Sorting falls out of scope

Thanks for good explanations. The City in the start would work only in case you always specify it… As you can’t use IN trick for cities as there are too many of them.
The problem with IN in such cases is generally related to sorting – results typically need to be sorted some way and as soon as you use IN on any key parts sorting can’t be done using index and filesort may be too slow for many matches. Though in reality problem can get even more complicated because sorting can be done by the function, for example by distance or using some sort of scoring system.

1) Looks like all fields are linked one-to-one to users so can be all stored in users table(may be except looking_for, but let’s keep it simple), some thing like (gender, age, available, looking_for, location, has_foto)
2) booleans – gender, has_foto; enums – available, looking_for, location; integer – age
3) All queries fall into X IN (…) AND Y IN (…) AND …

Some conditions fall into X = const – ref access that may be preferable over range. Condition on Age may have up to 60 values so it can be converted to IN instead of using BETWEEN [1].

As all boolean operators are ‘AND’, mysql can use either indexes built on multiple columns or intersection of multiple accesses by indexes. When we talk about index_merge_intersection, we should understand that mysql finds one set using an index, another set and then finds records which exist in both sets [2]. Consider this “age = 18 and available = ‘divorced’”. I’ve used Russia as target for investigation and here is results: both conditions – 58, divorced – 623, 18th – 3000 (looks like it’s limit and most probably it’s much more than that). It’s pretty obvious that using index (age, available) is much better all the time as it’s always more selective than any of its columns alone. If you don’t have such compound index then mysql most probably use index on “available” column only using ref access and after that filter by age. This is optimal choice. So in any case index intersection sucks. When it doesn’t suck? In something like “age = 68 AND available = ‘engaged’”, but even in this case compound index quickly returns a few rows.

Enough about intersections. About booleans. has_foto? How many of your users have a foto? 100%? 10%? Most probably it’s more than 70% or even close to 90%. So it’s very not selective column. Should we avoid using it in indexes? Yes, if it’s an index based on this column only. No, if it’s part of compound index. Why? Because more than 70% percents of people will mark “show people with fotos only” and mysql can use index to filter records before fetching any rows. How much it will give? Not many, worth investigation. May be impact will be negative especially when almost all people have fotos.

Gender? you can use it everywhere as peter suggested as 90% of searches will use one either ‘male’ or ‘female’ and it will give you twice smaller result set. However, you most probably should cheat a little and give mysql a hint by using “gender IN (‘male’,'female’)” when people looking for any gender.

Other. Location is mandatory with quite good selectivity comparing with other enumns. Age as well is mandatory, but it can select few rows or many that depends.

Final set?

1) (City, Age, Gender) or (City, Gender, Age) – you should benchmark and check EXPLAINS which one is better for all combinations of (male, female, both) and different ranges on Age with high selectivity and low. At the end you can end up with one index, both or may be consider using (City, Age). Whatever you select to use can be used as prefix for other indexes cuz conditions on these fields are mandatory.

2) You’ve selected mandatory prefix, for example (City, Gender, Age) is the most effective. Drop it and instead create (City, Gender, Age, LookingFor) and (City, Gender, Age, Available). So mysql can choose between two which is more selective.

3) You can even use (City, Gender, Age, LookingFor, Available) – needs investigation.

Aaaa… so long indexes. Let’s consider you’re superb and have 1 billion of users. If you’re smart enough and read mysql performance related articles then you’ll store all those fields as enums or tiny int with dictionary (in your up or table). 4(id)+1+1+1+1+1+1 = 10 bytes per record => >=10GB for data. Each index will be between 5GB and size of data (for innodb). May be I’m wrong here, but I do think that size of index(City, Gender, Age) will be smaller than size of three (City), (Gender) and (Age). Considering quite low standalone selectivity of those columns, it’s up to you decide what to do.

1. http://www.mysqlperformanceblog.com/2006/08/10/using-union-to-implement-loose-index-scan-to-mysql/#comment-1695
2. http://dev.mysql.com/doc/refman/5.1/en/index-merge-intersection.html

I mean if you have Index (A,B,C,D,E) and MySQL can use (A,B,C) index only for this query. Considering it is selective enough (enough meaning query response time is acceptable) you’re good. Though as your data size growths the acceptable selectivity tends to grow as well. If you have 100000 profiles selectivity of 1/10 is often good enough as scanning through 10000 rows in memory is often acceptable. With 50.000.000 profiles this would not be the case.

With Nachos case if you have “WHERE GENDER=’M’ AND RACE=’B’ AND HAIR=’Blond’ and it is very selective index on (GENDER,RACE,HAIR) would work quite well to resolve it. Though you really need to be looking at your best queries while picking solutions and so your worst cardinaliy.

For cases like you mentioned – Dating site search I would rather go with Sphinx all together. It handles cases like this beautifully (even if you do not have any full text search) though if you want to stick to MySQL you’ve got to look at the cardinality for different columns and query pattern and handle it appropriately. For example gender may not be selective but specified in 99% of the cases which makes it good first column in the index.

I have cases when billions records are loaded daily so it no more excites me.

Speaking about index merge – indeed there are cases when index merge works when multiple column index does not however if they both work multiple column index are most likely to be the winner.

There are always exceptions. Even running with MySQL defaults can be good fit for particular case. I’m just saying this is very typical red flag thing. Sure you need to look in details to check if it is really the case but it is called red flag exactly because it shows there are the problem in say 90% of all cases. If you made deliberate choice going with multiple column indexes good for you – if you can make this deliberate choice you already stand away from the crowd