Is there a fast, accurate Highlighter for Lucene?

3
votes

I've been using the (Java) Highlighter for Lucene (in the Sandbox package) for some time. However, this isn't really very accurate when it comes to matching the correct terms in search results - it works well for simple queries, for example searching for two separate words will highlight both code fragments in the results.

However, it doesn't act well with more complicated queries. In the simplest case, phrase queries such as "Stack Overflow" will match all occurrences of Stack or Overflow in the highlighting, which gives the impression to the user that it isn't working very well.

I tried applying the fix here but that came with a lot of performance caveats, and at the end of the day was just plain unusable. The performance is especially an issue on wildcard queries. This is due to the way that the highlighting works; instead of just working on the querystring and the text it parses it as Lucene would and then looks for all the matches that Lucene has made; unfortunately this means that for certain wildcard queries it can be looking for matches to 2000+ clauses on large documents, and it's simply not fast enough.

Is there any faster implementation of an accurate highlighter?

3
javalucene
The 'Highlighter for Lucene' link is currently broken. – pro
Thanks for pointing that out, I've fixed the link. – Mat Mannion
Now that Lucene 3.0.0 is out, you should find that upgrading everything to that will make things just work. HOWEVER - beware that the highlighter now has a dependency on lucene-memory from contrib as well (this will only show up when highlighting for exact matches) – Mat Mannion

3 Answers

2
votes

There is a new faster highlighter (needs to be patched in but will be part of release 2.9)

https://issues.apache.org/jira/browse/LUCENE-1522

and a back-reference to this question

1
votes

You could look into using Solr. http://lucene.apache.org/solr

Solr is a sort of generic search application that uses Lucene and supports highlighting. It's possible that the highlighting in Solr is usable as an API outside of Solr. You could also look at how Solr does it for inspiration.

1
votes

I've been reading on the subject and came across spanQuery which would return to you the span of the matched term or terms in the field that matched.