Thinking sphinx returning bad results (many documents missing)

Question

I am developing Ruby on Rails application which uses Thinking Sphinx. Unfortunately, from time to time (few times per month) search tends to return bad results (many documents missing). Reindexing helps, but this is not a solution for production.

I am experiencing bad results even when I am typing simple queries into rails console (like ThinkingSphinx.search 'skalee'). Sphinx search tool returns proper results, so indexing apparently works properly.

When I type in ThinkingSphinx.search('skalee').results[:words] I see proper numbers of hits (for example, term found in 30 documents) but ThinkingSphinx.search('skalee').results[:matches] contains, let's say, 2 documents. The numbers in results[:words] are equal to those I am getting with search.

I am using delayed delta but this problem is appearing even when I am not running ts:dd.

Answer 1

Just happened to stumble across this:

http://freelancing-god.github.com/ts/en/common_issues.html#deltas

Perhaps your user permissions are off?

Answer 2

Thinking Sphinx (or Delayed Delta, I don't remember well) adds special, internal attribute (sphinx_deleted or something like this) to all models. It is used to filter out destroyed records. Unfortunately, it works bad from time to time. After modifying the gem (getting rid of this attribute) everything works fine. Of course I need to wait till full reindexing (which I perform every night) to remove destroyed records from indexes, but this is minor disadvantage in my case. Alternatively, I could use Sphinx's kill list feature to filter out removed entries.