Why does idf value vary for the same term in multiple queried documents in Elasticsearch?

Question

I am executing a query on a ES index I made. The query consists of two terms and in the list of documents returned, I see one document ranked higher than another. This is despite the fact that the first document contains just one of the two terms, while the latter contains both.

I used the Explain API to see how the scores are being generated, and I found that for the same term "temperature", the idfs are being calculated differently. This is not expected because the idf depends only on the number of docs that contain the term and total docs in the index. Further, my index is not being changed at any time.

These are excerpts from the output of the two Explain calls. These contain the idfs :

Doc1:

"description": 
 "weight(description:temperatur in 0) [PerFieldSimilarity], result of:",
                            "details": [
                              {
                                "value": 0.7007727,
                                "description": "score(doc=0,freq=1.0), product of:",
                                "details": [
                                  {
                                    "value": 0.44917533,
                                    "description": "queryWeight, product of:",
                                    "details": [
                                      {
                                        "value": 3.1202636,
                                        "description": "idf(docFreq=5, maxDocs=50)",
                                        "details": []
                                      },
                                      {
                                        "value": 0.14395429,
                                        "description": "queryNorm",
                                        "details": []
                                      }
                                    ]

Doc2:

 "description": "weight(description:temperatur in 21) [PerFieldSimilarity], result of:",
                        "details": [
                          {
                            "value": 1.6804222,
                            "description": "score(doc=21,freq=1.0), product of:",
                            "details": [
                              {
                                "value": 0.61089635,
                                "description": "queryWeight, product of:",
                                "details": [
                                  {
                                    "value": 4.4011974,
                                    "description": "idf(docFreq=1, maxDocs=60)",
                                    "details": []
                                  },
                                  {
                                    "value": 0.1388023,
                                    "description": "queryNorm",
                                    "details": []
                                  }
                                ]

Why are the idfs different?

femtoRgon femtoRgon · Accepted Answer · 2016-05-24T22:35:13

I would guess you have more than one shard, right?

The idf will be calculated within that document's shard, not over the entire index. You can force it to calculate a global idf instead by adding ?search_type=dfs_query_then_fetch. For performance, it is recommended you don't use that in production.

For a more thorough explanation, see: Relevance is Broken!

Why does idf value vary for the same term in multiple queried documents in Elasticsearch?

1 Answers