Comments on: Speeding up GROUP BY if you want aproximate results

By: aboyon blog » Cosas no tan ciertas de CRC32() en MySQL

aboyon blog » Cosas no tan ciertas de CRC32() en MySQL — Tue, 03 Mar 2009 13:26:22 +0000

[...] otro dia leyendo el blog de MySQL Performance (MySQL Performance Blog’s) leia que hablaban del uso de CRC32() + BINARY en la consultas, y [...]

By: Benni

Benni — Thu, 04 Sep 2008 12:15:32 +0000

Hi all,

I wonder if this procedure would seed up a very slow query in my application: There is a value ‘x’ in each entry in my database between 0.0 and 100.0 (float). I want to divide this value into N “bins” and make mysql count how many entries are in each bin. My approach for e.g. 2 bins would be an INTERVAL query:

SELECT INTERVAL(x,50,101), count(*) FROM table GROUP BY 1;

Problem is: mysql can’t use an index on x because of the INTERVAL function. Is there a way I can use BINARY CRC32() to speed things up? Are there better ways to solve this issue than using an INTERVAL?

Thank you!

Benni

By: peter

peter — Wed, 09 Apr 2008 21:07:27 +0000

Upgrade to latest 5.0

simply CRC32(tag) should work.

By: Andrew Sutherland

Andrew Sutherland — Wed, 09 Apr 2008 20:42:32 +0000

MySQL 5.0.41

BINARY crc32(tag) is still much faster than tag. Is there a different way I can cast it to work with integers possibly?

By: peter

peter — Wed, 09 Apr 2008 20:37:42 +0000

CRC32 in PHP is crazy in the sense it gives results on 32bit and 64bit platforms…

In your case casting as BINARY you should get string which is slower. Typically just CRC32 should work. Your high collision rate was because it was running into the wall with max signed integer. What MySQL version is this ?

By: Andrew Sutherland

Andrew Sutherland — Wed, 09 Apr 2008 20:04:37 +0000

Hmm – I initially thought it was signed, because PHP’s crc32 is signed. That is interesting though, the mysql docs definitely say unsigned. Do you think there’s any danger in using my method? I haven’t found any collisions yet.

By: peter

peter — Wed, 09 Apr 2008 20:00:16 +0000

Andrew,

Something wrong with your query result – note 2147483647 value for CRC32 for few columns – this is MAX value for SIGNED int while CRC32 should be unsigned. Also for BINARY CRC32 you can get above this value. You might have found a bug in MySQL

By: Andrew Sutherland

Andrew Sutherland — Wed, 09 Apr 2008 19:49:35 +0000

I've been playing with crc32 and saw some very high collision rates, so I investigated a bit further and discovered GROUP BY was only taking 2^32 as a max. If I cast crc32 to BINARY though, my results worked perfectly. mysql> SELECT tag, COUNT(*) AS count, crc32(tag), BINARY crc32(tag) -> FROM tags -> GROUP BY BINARY crc32(tag) -> ORDER BY count DESC -> LIMIT 10 -> ; +------------+-------+------------+-------------------+ | tag | count | crc32(tag) | BINARY crc32(tag) | +------------+-------+------------+-------------------+ | spanish | 4576 | 874050868 | 874050868 | | vocab | 4103 | 1178479308 | 1178479308 | | vocabulary | 2786 | 2147483647 | 2425997691 | | french | 2247 | 2147483647 | 2943733342 | | english | 2087 | 746783232 | 746783232 | | science | 1957 | 1729573288 | 1729573288 | | latin | 1411 | 1421320458 | 1421320458 | | chapter | 1274 | 2147483647 | 4186027310 | | history | 1171 | 666529867 | 666529867 | | words | 939 | 1904025228 | 1904025228 | +------------+-------+------------+-------------------+ 10 rows in set (0.32 sec) Notice that both chapter, vocabulary, and french come up with the same crc32 values, but different BINARY crc32 values. So if you GROUP BY with regular crc32, those will group together. Thanks for the initial tip though, it's made a 30 second query into a .32 second query.

By: Log Buffer #88: a Carnival of the Vanities for DBAs

Log Buffer #88: a Carnival of the Vanities for DBAs — Fri, 14 Mar 2008 16:55:54 +0000

[...] the MySQL Performance Blog, Peter Zaitsev advises on speeding up GROUP BY if you want approximate results, which you might if accuracy is expensive and unnecessary when all you want is, “close enough [...]

By: peter

peter — Tue, 11 Mar 2008 02:05:44 +0000

Yes indeed order by null can be help for group by queries. How good depends on the group by execution mode and amount of groups you get in result set.