Adjust Lucene search result score by weight particular fields with same name

Question

I'm currently using Lucene as our full text search engine. But we need sorting the search result according to a particular field.

For example, if we have the following three documents in our index with exactly contents excepts the id field.

    val document01 = new Document()
    val field0100 = new Field("id", "1", Field.Store.YES, Field.Index.ANALYZED)
    val field0101 = new Field("contents", "This is a test: Linux", Field.Store.YES, Field.Index.ANALYZED)
    val field0102 = new Field("contents", "This is a test: Windows", Field.Store.YES, Field.Index.ANALYZED)
    document01.add(field0100)
    document01.add(field0101)
    document01.add(field0102)

    val document02 = new Document()
    val field0200 = new Field("id", "2", Field.Store.YES, Field.Index.ANALYZED)
    val field0201 = new Field("contents", "This is a test: Linux", Field.Store.YES, Field.Index.ANALYZED)
    val field0202 = new Field("contents", "This is a test: Windows", Field.Store.YES, Field.Index.ANALYZED)
    document02.add(field0200)
    document02.add(field0201)
    document02.add(field0202)

    val document03 = new Document()
    val field0300 = new Field("id", "3", Field.Store.YES, Field.Index.ANALYZED)
    val field0301 = new Field("contents", "This is a test: Linux", Field.Store.YES, Field.Index.ANALYZED)
    val field0302 = new Field("contents", "This is a test: Windows", Field.Store.YES, Field.Index.ANALYZED)
    document03.add(field0300)
    document03.add(field0301)
    document03.add(field0302)

Now, when I search Linux using IndexSearcher, I got the following result:

Document<stored,indexed,tokenized<id:1> stored,indexed,tokenized<contents:This is a test: Linux> stored,indexed,tokenized<contents:This is a test: Windows>>
Document<stored,indexed,tokenized<id:2> stored,indexed,tokenized<contents:This is a test: Linux> stored,indexed,tokenized<contents:This is a test: Windows>>
Document<stored,indexed,tokenized<id:3> stored,indexed,tokenized<contents:This is a test: Linux> stored,indexed,tokenized<contents:This is a test: Windows>>

When I search Windows, I get same result with same ordering.

Document<stored,indexed,tokenized<id:1> stored,indexed,tokenized<contents:This is a test: Linux> stored,indexed,tokenized<contents:This is a test: Windows>>
Document<stored,indexed,tokenized<id:2> stored,indexed,tokenized<contents:This is a test: Linux> stored,indexed,tokenized<contents:This is a test: Windows>>
Document<stored,indexed,tokenized<id:3> stored,indexed,tokenized<contents:This is a test: Linux> stored,indexed,tokenized<contents:This is a test: Windows>>

The question is that is it possible weight a particular fields when building index? For example, I would like make field0201 has higher score if its been matched when search.

In other words, when I search Linux, I would like get the result in the following order:

Document<stored,indexed,tokenized<id:2> stored,indexed,tokenized<contents:This is a test: Linux> stored,indexed,tokenized<contents:This is a test: Windows>>
Document<stored,indexed,tokenized<id:1> stored,indexed,tokenized<contents:This is a test: Linux> stored,indexed,tokenized<contents:This is a test: Windows>>
Document<stored,indexed,tokenized<id:3> stored,indexed,tokenized<contents:This is a test: Linux> stored,indexed,tokenized<contents:This is a test: Windows>>

And when I search for Windows, it still remains the original ordering, like the following:

Document<stored,indexed,tokenized<id:1> stored,indexed,tokenized<contents:This is a test: Linux> stored,indexed,tokenized<contents:This is a test: Windows>>
Document<stored,indexed,tokenized<id:2> stored,indexed,tokenized<contents:This is a test: Linux> stored,indexed,tokenized<contents:This is a test: Windows>>
Document<stored,indexed,tokenized<id:3> stored,indexed,tokenized<contents:This is a test: Linux> stored,indexed,tokenized<contents:This is a test: Windows>>

I tried using field0201.setBoost(), but it will change the ordering of search result both when I search Linux or Windows.

It looks like the documents all contain the same data except for the id. Why would you have an expectation that the scores are different? — huynhjl
@huynhjil Because the contents come from different source. I would like a field come from particular source have higher score if it is matched with search terms. In other words, it should sort with the using (score lucene calculated, the source of field) pair. — Brian Hsu
Would it be possible for you to use the Sort instance that you pass to a TopFieldCollector to do your sorting? ...or do you explicitly want to do it with the score of your fields (This would only work if the contents are not the same)? — csupnig
@csupning, I'm think that I TopFieldCollector is not feasible solution. Since I could not find a way to store extra information to a field, so I could not sort by the extra information. What I need to achieve is when the search result hit two field has same content but come from different source(ex, one from source A, one from source B), the field come from source B will have higher score. But if the field is not matched with search request, then it should not affect the ordering of search result. — Brian Hsu

huynhjl huynhjl · Accepted Answer · 2011-04-14T11:49:41

I think it should be possible if you put your data for different sources in fields with different names. You can set a boost at index time, but if you use the same name I think the boost would apply to all fields with the same name - based on the setBoost javadoc. So if you do this instead:

val field0201 = new Field("content-high", "This is a test: Linux", ...)
field0201.setBoost(1.5f)
val field0202 = new Field("content-low", "This is a test: Windows", ...)

And then query with content-high:Linux content-low:Linux (using a boolean query with two should clauses both set to term Linux), then the boost for content-high should increase the document score if the match is in that field. Use explain to see whether that works.

Adjust Lucene search result score by weight particular fields with same name

1 Answers