Sphinx 0.9.9 is finally here.

peter — Fri, 04 Dec 2009 19:37:18 +0000

Sphinx 0.9.9 is finally released to General Availability. The previous version 0.9.9-rc2 was released back in April so there was quite a gap. This release fixes about 40 bugs which are mostly rare or insignificant. Fixing bugs in 0.9.9 was smaller portion of work done in Sphinx during last half a year – 0.9.9-rc2 was quite scale and in fact a lot of users were using 0.9.9-x branch instead of old 0.9.8 branch in production. Most of work was done on new features in 0.9.10 as well as live updates.

Entry posted by peter | No comment

Add to: | | | |

Talking MySQL to Sphinx

peter — Mon, 20 Apr 2009 05:34:27 +0000

In the recently released Sphinx version 0.9.9-rc2 there is a support for MySQL wire protocol and SphinxQL - SQL-like language to query Sphinx indexes. This support is currently in its early preview stage but it is still fun to play with.

A thing to mention - unlike MySQL Storage Engines, some of which as InfoBright or KickFire take over execution after parsing, Sphinx MySQL support has nothing to do with MySQL - it is implementation of the wire protocol from scratch.

For this test I was not interesting in the full text search performance, we already know Sphinx is much faster than MySQL build in full text search. I was rather interested to look performance of other queries, not using Full Text Search.

PLAIN TEXT

SQL:

[root@r27 sp]# mysql --host 127.0.0.1 --port 3307
Welcome TO the MySQL monitor. Commands end WITH ; OR \g.
Your MySQL connection id IS 1
Server version: 0.9.9-id64-rc2 (r1785)
Type 'help;' OR '\h' FOR help. Type '\c' TO clear the buffer.

For the tests I used the table from the forum search engine, leaving just bunch of ids in it, removing everything else:

PLAIN TEXT

SQL:

CREATE TABLE `sptest` (
`id` bigint(20) UNSIGNED NOT NULL,
`site_id` int(10) UNSIGNED NOT NULL,
`forum_id` int(10) UNSIGNED NOT NULL,
`author_id` int(10) UNSIGNED NOT NULL,
`num_links` smallint(5) UNSIGNED NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=utf8

This table contained some 25 millions of rows and no indexes there defined - Sphinx does not support explicit indexes and it is clear when you can use index for sort MySQL will be a lot faster.

First - Sorting. Sphinx is smart doing sorting because it does not try to sort everything but if you ask but rather only number of rows it needs to reach the LIMIT

Sphinx

PLAIN TEXT

SQL:

mysql> SELECT forum_id AS f FROM sptest ORDER BY author_id DESC LIMIT 10;
+------------+--------+----------+
| id | weight | forum_id |
+------------+--------+----------+
| 6739362135 | 1 | 2736983 |
| 6739362391 | 1 | 2736983 |
| 6739338327 | 1 | 1024599 |
| 6739357527 | 1 | 1023063 |
| 6739359063 | 1 | 1024599 |
| 6739305559 | 1 | 2558807 |
| 6739336791 | 1 | 2558807 |
| 6739300695 | 1 | 208215 |
| 6739297111 | 1 | 2736471 |
| 6739296855 | 1 | 2736471 |
+------------+--------+----------+
10 rows IN SET (7.92 sec)

MySQL

PLAIN TEXT

SQL:

mysql> SELECT forum_id AS f FROM sptest ORDER BY author_id DESC LIMIT 10;
+---------+
| f |
+---------+
| 2736983 |
| 2736983 |
| 1024599 |
| 1023063 |
| 1024599 |
| 2558807 |
| 2558807 |
| 208215 |
| 2736471 |
| 2736471 |
+---------+
10 rows IN SET (17.91 sec)

As you can see Sphinx adds couple of extra columns to result set even if you have not asked it.

Another thing to try is GROUP BY - Sphinx executes GROUP BY in fixed memory which means results may be approximate - this is geared towards full text search applications when exact number is not important.

Sphinx

PLAIN TEXT

SQL:

mysql> SELECT max(forum_id) AS m,author_id AS a FROM sptest GROUP BY author_id ORDER BY m DESC LIMIT 10;
+------------+--------+----------+-----------+---------+
| id | weight | forum_id | author_id | m |
+------------+--------+----------+-----------+---------+
| 6739362135 | 1 | 2736983 | 139452247 | 2736983 |
| 6738995287 | 1 | 1762135 | 134125655 | 2736727 |
| 6739296855 | 1 | 2736471 | 139450967 | 2736471 |
| 6739297111 | 1 | 2736471 | 139451223 | 2736471 |
| 6739227479 | 1 | 2736215 | 139449687 | 2736215 |
| 6739227735 | 1 | 2736215 | 139449943 | 2736215 |
| 6739226967 | 1 | 2735959 | 139449175 | 2735959 |
| 6739227223 | 1 | 2735959 | 139449431 | 2735959 |
| 6739223383 | 1 | 2735703 | 139448663 | 2735703 |
| 6739223639 | 1 | 2735703 | 139448919 | 2735703 |
+------------+--------+----------+-----------+---------+
10 rows IN SET (32.47 sec)

MySQL

PLAIN TEXT

SQL:

mysql> SELECT max(forum_id) AS m,author_id AS a FROM sptest GROUP BY author_id ORDER BY m DESC LIMIT 10;
+---------+-----------+
| m | a |
+---------+-----------+
| 2736983 | 139452247 |
| 2736727 | 134125655 |
| 2736471 | 139450967 |
| 2736471 | 139451223 |
| 2736215 | 139449687 |
| 2736215 | 139449943 |
| 2735959 | 139449175 |
| 2735959 | 139449431 |
| 2735703 | 139448663 |
| 2735703 | 139448919 |
+---------+-----------+
10 rows IN SET (1 min 15.03 sec)

Another optimization I wanted to check is the "early block reject" which should allow to quickly throw away large blocks of attributes if they do not contain any data:

Sphinx

PLAIN TEXT

SQL:

SELECT max(author_id) AS a ,forum_id AS f FROM sptest WHERE num_links=1;
Empty SET (2.70 sec)

MySQL

PLAIN TEXT

SQL:

mysql> SELECT max(author_id) AS a ,forum_id AS f FROM sptest WHERE num_links=1;
+------+---+
| a | f |
+------+---+
| NULL | NULL |
+------+---+
1 row IN SET (4.29 sec)

I would expect much larger lead in this case because of this optimization but it seems to be broken in the tested version.

Also note the result set difference - Sphinx finds no rows and creates no groups while MySQL reports NULL group as a result.

SphinxQL at this point is rather picky - it wants AS for all the expressions, it also could not parse some queries for no reason though I expect these things to be polished in the near future. The good thing is the query execution maps to the same execution engine which is quite stable which means it will likely stabilize soon.

Sphinx also offers number of extensions to the SQL which are helpful for search use cases - WITHIN GROUP ORDER BY allows to select which item to pick within given group (like if you want to show most recent document, or most relevant) and others.

You might find using Native API more feature full at this point but command line language is very helpful for testing and debugging purposes as well as so Sphinx can be accessed from languages which doe not have native Sphinx API implemented - everyone seems to be able to talk to MySQL these days.

Now on performance - for given class of queries Sphinx was just 1.5-2 times faster. I honestly hoped for more, though I carefully picked queries which are reasonably good for both of them - it is easy to "break" MySQL making it to do group by with on disk temporary table which will make Sphinx much faster and few others.

The true gain from Sphinx however comes from its ability to scale almost linearly using multiple CPU cores and multiple nodes in the system. The raw scan speed was almost 10 millions of rows per second (this is on rather outdated CPU I used for testing) - this means you should be able to scan through 100M+ rows on the single modern 8 core server which is quite a number.

Entry posted by peter | 4 comments

Add to: | | | |

Percona’s patches spread to a wider audience

Baron Schwartz — Tue, 14 Oct 2008 03:50:00 +0000

Percona's patches are now available to a wider audience via OurDelta, a community effort to provide builds with features (Percona patches, Google patches, etc) and storage engines (PBXT, Sphinx, etc) that aren't in the main MySQL server. Arjen Lentz is really the brainchild behind this. Kudos Arjen!

What does this mean for the Percona patches? Well, now you can get them in more places. But it doesn't change our own commitment to keep innovating in ways our customers (and we ourselves) find useful. We're still building our own builds and hosting the downloads from our own website; it's important that you can get a build that you know authoritatively came from us and was built by us. The Percona patches and builds represent our best judgment about what's the state of the art in the MySQL server for real-world usage, and we have no plans to change that.

The OurDelta project is a great way to get pre-built versions of the server with cutting-edge features like PBXT. This should help lower the barriers to trying these things out. The Percona builds and patches are conservative (except for the -highperf builds, which are experimental) and won't include such features, at least not for the foreseeable future. We're really glad that others are adopting our patches and blending them with features our builds won't include. OurDelta also has a build infrastructure behind it, which is quite a nice thing to have, and we expect that bug reports and patches may even flow "upstream" back to us.

Of course the project is very new, and many changes will undoubtedly happen in the weeks and months to come. But you have to start somewhere, and it's really nice to see such an initiative.

Entry posted by Baron Schwartz | 7 comments

Add to: | | | |

MySQL Performance Blog » sphinx

Sphinx 0.9.9 is finally here.

Talking MySQL to Sphinx

Percona’s patches spread to a wider audience