Elastic search ---: MUST_NOT query not working

Question

I have a query in which i want to add a must_not clause that would discard all records that have blank data for a some field. I tried a lot of ways but none worked. when I issue the same query (mentioned below) with other specific fields then it works fine.

this query should get all records that do not have "registrationType1" field empty/blank

query:
{
"size": 20,
"_source": [
"registrationType1"
],
"query": {
"bool": {
"must_not": [
{
"term": {
"registrationType1": ""
}
}
]
}
}
}

the results below still contains "registrationType1" with empty values

results:

**"_source": {
"registrationType1": ""}}
, * {
"_index": "oh_animal",
"_type": "animals",
"_id": "3842002",
"_score": 1,
"_source": {
"registrationType1": "A&R"}}
, * {
"_index": "oh_animal",
"_type": "animals",
"_id": "3842033",
"_score": 1,
"_source": {
"registrationType1": "AMHA"}}
, * {
"_index": "oh_animal",
"_type": "animals",
"_id": "3842213",
"_score": 1,
"_source": {
"registrationType1": "AMHA"}}
, * {
"_index": "oh_animal",
"_type": "animals",
"_id": "3842963",
"_score": 1,
"_source": {
"registrationType1": ""}}
, * {
"_index": "oh_animal",
"_type": "animals",
"_id": "3869063",
"_score": 1,
"_source": {
"registrationType1": ""}}**

PFB mappings for the field above

"registrationType1": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword"
}
}
}

Answer 1

You need to use the keyword subfield in order to do this:

{
  "size": 20,
  "_source": [
    "registrationType1"
  ],
  "query": {
    "bool": {
      "must_not": [
        {
          "term": {
            "registrationType1.keyword": ""       <-- change this
          }
        }
      ]
    }
  }
}

Answer 2

If you do not specify any text value on the text fields, there is basically nothing to analyze and return the documents accordingly.

In similar way, if you remove must_not and replace it with must, it would show empty results.

What you can do is, looking at your mapping, query must_not on keyword field. Keyword fields won't be analysed and in that way your query would return the results as you expect.

Query

POST myemptyindex/_search
{
  "query": {
    "bool": {
      "must_not": [
        {
          "term": {
            "registrationType1.keyword": ""
          }
        }
      ]
    }
  }
}

Hope this helps!

Answer 3

I am using elasticsearch version 7.2, I replicated your data and ingested in my elastic index,and tried querying with and without .keyword.

I am getting the desired result when using the ".keyword" in the field name.It is not returning the docs which have registrationType1="".

Note - The query does not works when not using the ".keyword"

I have added my sample code below, have a look if that helps.

from elasticsearch import Elasticsearch
es = Elasticsearch()
es.indices.create(index="test", ignore=400, body={
"mappings": {
    "_doc": {
        "properties": {
            "registrationType1": {
                "type": "text",
                "field": {
                    "keyword": {
                        "type": "keyword"
                    }
                }
            }
        }
    }
}
})

data = {
"registrationType1": ""
}

es.index(index="test",doc_type="_doc",body=data,id=1)
search = es.search(index="test", body={
"size": 20,
"_source": [
    "registrationType1"
],
"query": {
    "bool": {
    "must_not": [
        {
            "term": {
                "registrationType1.keyword": ""
            }
        }
    ]
    }
}
})

print(search)

Executing the above should not return any results as we are inserting empty for the field

Answer 4

There was some issue with the mappings itself, I deleted the index and re-indexed it with new mappings and its working now.