Because CRC32 is just 32bits 4bytes compressed url would take much longer.

Of course if you can sue some simple compression like you need to store image URLs and so would store 123 instead of img.site.com/123.jpg but for general urls you will not compress them even close to this number

I do not understand you. You can perfectly check EXACTLY if URL exists or not using plain SQL statement. Simply only CRC part will be checked by index and post-checking will be done after full row is read.

Now planning your database structure you plan it for certain queries. If you need to serve arbitrary queries you design schema appropriately. If you only need some queries but you need them really fast you design schema differently.

Think about it in car terms. If you only want to entertain your girlfriend and get yourself to work 2 seater sportscar may do good job for you. If you may need to load 7 people or load a sofa Minivan works better. So what ? You can’t say one or another is better without knowing the purpose.

With domain table you propose certain way of normalization which is good for some applications again.

For some workloads prefix indexes may work or other index structures.

Speaking about 1-10b rows is a prototype size so we’re rather picky on optimizing queries.

What if your boss comes to you and asks you : hey dude, how many URL’s do you have in your database that points to the website http://members.aye.net/

For sure you HAVE to do something like this SELECT * FROM table WHERE url LIKE (’http://members.aye.net/%’);

What I would do is may be create a domain table to store domains only based on CRC32,
then create 26 tables for your URL’s and partition your URI’s based on the URI

so table domain stores : http://www.dell.com, http://www.vantwisk.nl, http://www.mysqlperformanceblog,com
then table URI stores : 2008/04/04/multi-column-in-clause-unexpected-mysql-issue/#more-361, blabla.html, myblog/entry/bla.html

On the doamin table you can create an idnex by CRC, but I would just use the.
On table URI you simple create an index on the first XX characters to keep index size low.

Data retrieval is done using a couple of stored procedures to get the right data and filter any duplicates.

Ries
PS: you could partition your tables if you need more. Since you didn’t mention how many rows you ant to store it’s hard to guess….
You did mention massive, how much is massive for you??? in the order of 1000mil records???

Ries

It has to do with IN, your query works fine.

mysql> EXPLAIN SELECT url FROM 124pages.124pages WHERE (url_crc,url) = (484036220,’http://www.dell.com/’);
+—-+————-+———-+——+—————+———+———+——-+——+————-+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+—-+————-+———-+——+—————+———+———+——-+——+————-+
| 1 | SIMPLE | 124pages | ref | url_crc | url_crc | 4 | const | 1 | Using where |
+—-+————-+———-+——+—————+———+———+——-+——+————-+
1 row in set (0.03 sec)

I know it shouldn’t matter, but have you tried:

EXPLAIN SELECT url FROM 124pages.124pages WHERE (url_crc,url) = (484036220,’http://www.dell.com/’);

to see if it is due to IN or rather due to multiple columns?

If there is a difference, it seems likely that it is a bug, which may turn out to be trivial to fix.

I do not think prepared statements will be faster if you’re doing one by one checks - the savings on query parsing are much smaller than loss by having many roundtrips.

If you would need to check 1000000 urls such a way you can use prepared statements together with batches and check them by 1000 or something like that. This likely would be the most efficient approach.

In this case it is easy enough to get the query which works. There are many cases when you can’t get MySQL to do what you want - consider for example bunch of subqueries, in these cases I would handle thing on the application - ie do SELECT create IN list and generate second query.

May be UNION ALL is more native representation.

Also, wonder why mysql even doesn’t consider index merge.

As in many other cases the best things is to make tests against your own data