Elastic query returning incorrect results

Question

Here is my query ....

{
  "size": 100,
  "query": {
    "bool": {
      "filter": {
        "bool": {
          "should": [
            {
              "terms": {
                "dkncourseintakes": [
                  "Trimester 3"
                ]
              }
            },
            {
              "range": {
                "dkncourseatarcloud": {
                  "lte": "100"
                }
              }
            },
            {
              "range": {
                "dkncourseatarwaurnponds": {
                  "lte": "100"
                }
              }
            },
            {
              "range": {
                "dkncourseatarwarrnambool": {
                  "lte": "100"
                }
              }
            },
            {
              "range": {
                "dkncourseatarburwood": {
                  "lte": "100"
                }
              }
            },
            {
              "range": {
                "dkncourseatarwaterfront": {
                  "lte": "100"
                }
              }
            }
          ],
          "must": [
            {
              "terms": {
                "dkncoursequal": [
                  "under_bachelor_degree"
                ]
              }
            }
          ]
        }
      },
      "must": [
        {
          "match": {
            "dkncoursestudent": "Domestic"
          }
        },
        {
          "terms": {
            "dknpagetagia": [
              "ia-Business"
            ]
          }
        },
        {
          "terms": {
            "dkncourselocations": [
              "Burwood"
            ]
          }
        }
      ]
    }
  }
}

One of the results returned by this query is this ... As you can see my query clearly search for courses in "Trimester 3", I am puzzled why this course with "Trimester 1" is returned ?

{
        "_index": "all_courses",
        "_type": "courses",
        "_id": "03c154fe7ee8029472533222340369d43d4fbff5",
        "_score": 2.030092,
        "_source": {
          "dkncoursetype": "Undergraduate",
          "dkncoursequal": "under_bachelor_degree",
          "dkncourselocations": "Burwood",
          "dkncourseatarwarrnambool": 0,
          "dknpagetagia": "ia-Business,ia-English_and_International_Languages",
          "dkncourseatarburwood": 65,
          "dknpagetagfands": "Faculty_of_Arts_and_Education",
          "dkncourseatarwaterfront": 0,
          "dkncoursestudent": "Domestic",
          "dkncourseintakes": "Trimester 1" <=== THIS IS THE PROBLEM
        }

If I change/tweak the query from this

"bool": {
              "should": [
                {
                  "terms": {
                    "dkncourseintakes": [
                      "Trimester 3"
                    ]
                  }

to this

 "bool": {
          "must": [
            {
              "terms": {
                "dkncourseintakes": [
                  "Trimester 3"
                ]
              }
            }

I get no results.

Also, I am using this plugin to generate request body.

And my query looks like this ..

 var body = bodybuilder()
                    //.filter('terms', 'dkncourseintakes', intake_checkbox_value)
                    .query('match', 'dkncoursestudent', student_radio_value)
                    .query('terms', 'dknpagetagia', interest_checkbox_value)
                    .query('terms', 'dkncourselocations', location_checkbox_value)
                    .filter('terms', 'dkncourseintakes', intake_checkbox_value)

                    .orFilter('range', 'dkncourseatarcloud', {lte: atar_radio_value})
                    .orFilter('range', 'dkncourseatarwaurnponds', {lte: atar_radio_value})
                    .orFilter('range', 'dkncourseatarwarrnambool', {lte: atar_radio_value})
                    .orFilter('range', 'dkncourseatarburwood', {lte: atar_radio_value})
                    .orFilter('range', 'dkncourseatarwaterfront', {lte: atar_radio_value})

                .filter('terms', 'dkncoursequal', qual_checkbox_value)
                    .size(100)
                    .build()

Here is my mapping:

{
        "settings": {

            "number_of_shards": "1",
            "number_of_replicas": "0",
            "analysis": {
                "analyzer": {
                    "semi_colon_analyzer": {
                        "type": "pattern",
                        "pattern": ";",
                        "lowercase": false
                    },
                    "comma_analyzer": {
                        "type": "pattern",
                        "pattern": ",",
                        "lowercase": false
                    }
                }

            }
        },
        "mappings": {
            "test": {
                "properties": {     
                "dkncoursestudent": {
                        "type": "text"
                    },
                    "dknpagetagia": {
                        "type": "text",
                        "analyzer": "comma_analyzer",
                        "search_analyzer": "comma_analyzer"
                    },      
                    "dkncoursequal": {
                        "type": "text"
                    },
                    "dkncourseatar": {
                        "type": "integer"
                    },  
                    "dkncourseintakes": {
                        "type": "text",
                        "analyzer": "semi_colon_analyzer",
                        "search_analyzer": "semi_colon_analyzer"
                    },
                    "dkncourselocations": {
                        "type": "text",
                        "analyzer": "semi_colon_analyzer",
                        "search_analyzer": "semi_colon_analyzer"
                    }
                }
            }
        }
    }

I have defined custom analyzers for these 3 fields

dkncourseintakes (indexed like "Trimester 1; Trimester 2; Trimester 3")
dkncourselocations (indexed like "Burwood; Cloud Campus")
dknpagetagia (indexed like "ia-Business,ia-English,ia-Humanities,ia-Law")

Also, can someone please advise how to confirm that a custom analyzer is used for searching ? i.e. 'semi_colon_analyzer' in my case

Thanks for reading thus far, I'll really appreciate if anyone can help me figure out how to make this work.

You have mentioned Trimester 3 in should block with other more conditions. Should act as OR.. range query for dkncourseatarburwood and dkncourseatarwarrnambool matches in this case. that's why it is being returned. — Richa
You want "dkncourseintakes": [ "Trimester 3" ] as mandatory condition ?? — Richa
@Richa: Yes, I know "should" act as OR and "must" act as "AND". So when I use "must", why do I get zero results ? Yes, I think I want dkncourseintakes": [ "Trimester 3" ] as a mandatory condition. But please keep in mind that this should also work if dkncourseintakes contain multiple values like dkncourseintakes": [ "Trimester 1", "Trimester 3" ] — Slyper
AND will act on all the conditions written inside MUST block.. All conditions including range and terms might not be fulfilled in any record — Richa

Slyper Slyper · Accepted Answer · 2017-05-30T13:38:55

Never mind guys. Looked like the mapping was incorrect .... It was like this ...

"semi_colon_analyzer": {
                        "type": "pattern",
                        "pattern": ";", <=== This was the problem
                        "lowercase": false
                    }

Changed it to "pattern": "; " (just added space after semi-colon, all good now)

Elastic query returning incorrect results

1 Answers