comparative grouping of large text fields - FULLTEXT Search, SOLR, or OTHER?

Question

I have about 150,000 LARGE TEXT fields inside of a mysql database. Each TEXT field represents about 1page of text and will be referred to as a page from here on out.

I would like to group these pages into small groups of very relevant pages. My thought on how to do this is to use FULLTEXT Index in mysql. I will input blocks of text and have the FULLTEXT search rank relevance to other pages. Is there a limit to how big of a text block you can put into FULLTEXT search?

Are there other ways to do this using PHP/mysql? Do I need to implement a search engine like SOLR?

Determining relevance is a hard problem. Do you have a list of categories/terms with which you want to classify the fields or are you looking to discover relevance from them? — Sukumar

symcbean symcbean · Accepted Answer · 2011-08-03T12:55:01

Although it's handy as a quick fix and for certain niche applications there are a number of issue with mysql's fulltext indexing, particularly around ranking and scalability. I'd recommend implementing a simple keyword lookup search engine

comparative grouping of large text fields - FULLTEXT Search, SOLR, or OTHER?

1 Answers