AFAIK there is still no way, but I’m not sure why people regard this as a problem. It wouldn’t be too hard to use a MyISAM table just for search, and keep the information via rules (functions, triggers, whatever). In a sense, if it’s not critical to keep the index 100% up-to-date, it’s better to cron it and rebuild at interval. In fact, MySQL documentation indicates it is much faster to rebuild sometimes as well. There’s some underlying algorithmic reason, I’m sure.

Also, I’ve noticed that it’s extremely expensive (others might benefit from this knowledge) to hit 2 FT indices at a time. Again, probably some underlying reason why this is not efficient. WHERE FT_QUERY_1 AND FT_QUERY_2 seems to be OK, but “OR” seems to be a huge problem on large datasets.

Therefore, it would seem that the solution to searching multiple FT indices is NOT to do so, and create a table or column that puts all the information together. Unfortunately, in MySQL, there are no expression-based indexes, but I’m not 100% sure that would work for FT.

Hope the above helps someone. Also, reasonable results can be achieved by using BOOLEAN queries to restrict, and expressions to sort. AFAIK, natural FTS can’t sort until all results are returned anyway.

I have reasonable success with MySQL 5.1 and 100k+ product rows. Sphinx is better in some ways, but FT is fairly mature now.

One thing I cannot figure out is how to efficiently understand/normalize the numbers returned from natural FTS. i.e. to add a boost from another non FTS column. The numbers are effectively random from one query to the next. It would be nice if MySQL had a way to rank from 1-100, but I’m not sure that’s a free operation. I’m not a DBMS expert. You can’t just add +5, though, and selecting MAX is not cheap.

If anyone knows how to do that, I’d love to know

Mysql Examples
http://mysqlexamples.blogspot.com

http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes

You can create a full-text database search service, return results as HTML/XML/JSON. It uses the Lucene directly in java, but can be easily used with Ruby, PHP, or any existing database web applicatoins.

You can easily index, re-index, incremental-index. It’s also highly scalable and easily customizable.

The best thing is, it’s super easy. You can create a production-level search in 3 minutes.

I’ve asked Andrew to add some kind of tutorial and he will do that once he finds a time. Honestly we do not use sphinx as storage engine but instead as stand alone application most of the time simply retrieving rows from MySQL based on IDs.

It makes it much easier in terms of profiling etc.

I think Sphinx is great solution for MySQL at this point especially if you’re looking for search performance and large data sizes.

In your main post you outline that sphinx can be used by MySQL 5.1 as storage engine?
Can you please explain this in detail?

I’m currently using MyISAM becasue of the fulltext index for two forums.
Size is 2 Million records and 100,000 records.
Average new rows per day 400.
Average page hits per day 250,000

I’m very unhappy about using MyISAM for obvious table locking reasons and issues.
The combination of complicated (long) search + insert + many parallel small selects gives terrible performance with MyISAM.

I would liek to switch to something “better”.
I wonder if switching to a mix of innodb and myisam could help.
Running the table under Innodb and having an extar fulltext ISAM table.
This way normal reads that don’t use FT search could run without slowdown from the normal table …

A FT search for Inno would propably the easiest way.
Or do you think that sphinx would a better solution?

Regards to you and vadim
Gunnar

Please post your question at forum, forum.mysql.com as it is not relevant to the blog post.

it’s just I can’t understand why it happens that whenever i try to run queries at the first time it takes lots of time while after running it at first time they don’t take much time to run…..i want to run my all queries run asmuch faster as they get run after running them first time….i hope you’ll get what i mean…….i’m hopeful for your favourable response.

I have not used Senna. It is however not full text engine which is embedded in MySQL, but I see there are some patches so it can be bound to MySQL.

Only fractions of web site are available in English so I could not find all info I was looking for. If you know author email let me know.