Sphinx search ranking broken?

Question

Has anyone ever gotten the Sphinx ranking options to work? I've read the manual and the book but cannot get ranking working at all. From what I understand, ranking simply computes the weights in a different manner, doesn't do any type of sorting. I have my results sorted by @weight (internal sphinx field) and using sort mode extended, which you need for this, yet cannot see any difference between different ranking modes. My config is something like this:

$cl->SetMatchMode( SPH_MATCH_EXTENDED2 );  
$cl->SetSortMode ( SPH_SORT_EXTENDED, "mylang DESC, @weight DESC, @id"); 

Neither of these makes any difference:

$cl->setRankingMode(SPH_RANK_SPH04);
$cl->setRankingMode(SPH_RANK_PROXIMITY_BM25);

And the weights are the same in either mode.

Ultimately, what I'm trying to achieve is to have terms that match exactly be sorted towards the top. So for example, if searching for "Harry Potter" the results should be as follows:

Harry Potter
Harry Potter and the potters
Harry Potter and the Prisoner of Azkaban
Harry Potter and the Deathly Hallows: Part 1

This is just an example, but the first result should be the one that contains the exact search term, then the others would follow. This is not happening. Anyone have any experience with this?

Answer 1

Do you have any other records in index except which start from "Harry Potter"? If no, then phrase "Harry Potter" will be penalized by ranking algorithm.

See my article about that: Interesting thing about BM25 in Sphinx Search

All of you records have exact match for "Harry Potter", so I suppose records with more words would ranked higher.

Solution could be to use attribute which store records size in bytes:

sql_query = select field, length(field) as f_size from ....

Attribute:

sql_attr_uint = f_size

Sphinx sort mode:

$cl->SetSortMode ( SPH_SORT_ATTR_ASC, 'f_size' );

Answer 2

Turns out that SPH_RANK_SPH04 is not included in the sphinxapi.php file in version 0.9.9!!! So even though you're calling it it's not taken into account and furthermore does not produce an error.

This is terrible because it makes it very hard to troubleshoot.

I've posted this as the answer in the hopes that it helps someone else. We lost almost 2 days going crazy over this until we figured it out.

Furthermore, there is a bug in 2.0.1 which doesn't really bring some exact matches to the front, for that you need 2.0.2 (which you need to get from SVN) or above, but I'd be very weary of using experimental versions in production.

Hopefully the Sphinx developers will take care of this soon.

PS Looking back at the developer diaries, it does say:

"As of 1.10-beta, Sphinx has 8 different rankers"

We upgraded from 0.9.9 to 2.0.1 and must have left the api file behind, and in desperation I never even checked this. It would still be nice for Sphinx to throw an error if the ranking mode doesn't exist (as it does for other modes such as matching), and the 2.0.1 bug is still there as far as we can tell in our tests.