Elasticsearch not working with 'not_analyzed' index

Question

I am unable to figure out why elasticsearch not searching with not_analysed indexes. I have following settings in my model,

settings index: { number_of_shards: 1 } do
      mappings dynamic: 'false' do
        indexes :id
        indexes :name, index: 'not_analyzed'
        indexes :email, index: 'not_analyzed'
        indexes :contact_number
      end
    end

    def as_indexed_json(options = {})
      as_json(only: [ :id, :name, :username, :user_type, :is_verified, :email, :contact_number ])
    end

And my mapping at elasticsearch is right, as below.

{
  "users-development" : {
    "mappings" : {
      "user" : {
        "dynamic" : "false",
        "properties" : {
          "contact_number" : {
            "type" : "string"
          },
          "email" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "id" : {
            "type" : "string"
          },
          "name" : {
            "type" : "string",
            "index" : "not_analyzed"
          }
        }
      }
    }
  }
}

But issue is when I make search on not analyzed fields (name and email, as I wanted them to be not analyzed) it only search on full word. Like in the example below it should have return John, Johny and Tiger, all 3 records. But it only returns 2 of the records.

I am searching as below

  settings = {
    query: {
      filtered: {
        filter: {
          bool: {
            must: [
              { terms: { name: [ "john", "tiger" ] } },
            ]
          }
        }
      }
    },
    size: 10
  }

  User.__elasticsearch__.search(settings).records

This is how I am creating index on my user object in callback after_save,

User.__elasticsearch__.client.indices.create(
                index: User.index_name,
                id: self.id,
                body: self.as_indexed_json,
              )

Some of the document that should match

[{
      "_index" : "users-development",
      "_type" : "user",
      "_id" : "670",
      "_score" : 1.0,
      "_source":{"id":670,"email":"[email protected]","name":"john baba","contact_number":null}
    },
    {
          "_index" : "users-development",
          "_type" : "user",
          "_id" : "671",
          "_score" : 1.0,
          "_source":{"id":671,"email":"[email protected]","name":"Johny Rocket","contact_number":null}
        }

    , {
          "_index" : "users-development",
          "_type" : "user",
          "_id" : "736",
          "_score" : 1.0,
          "_source":{"id":736,"email":"[email protected]","name":"tiger sherof", "contact_number":null}
        } ]

Any suggestions please.

how did you index? "John" or 'john'? could you show us the document that you think should match? — ChintanShah25
@ChintanShah25 thanks for the suggestion actually it is only search for partial word like if the names would be John Baba and Johny Rocket, and the query string is John -- It should have returned 2 records but it is only returning a single record i.e John. Any suggestions Thanks — Muhamamd Awais
looking at your documents, you wont get any results. What is your requirements? what should match and what should not? also how johny would match john? — ChintanShah25

ChintanShah25 ChintanShah25 · Accepted Answer · 2016-01-30T15:17:36

I think you would get desired results with keyword toknizer combined with lowercase filter rather than using not_analyzed.

The reason john* did not match Johny was due to case sensitivity. This setup will work

{
  "settings": {
    "analysis": {
      "analyzer": {
        "keyword_analyzer": {
          "type": "custom",
          "filter": [
            "lowercase"
          ],
          "tokenizer": "keyword"
        }
      }
    }
  },
  "mappings": {
    "my_type": {
      "properties": {
        "name": {
          "type": "string",
          "analyzer": "keyword_analyzer"
        }
      }
    }
  }
}

Now john* will match johny. You should be using multi-fields if you have various requirements. terms query for john wont give you john baba as inside inverted index there is no token as john. You could use standard analyzer on one field and keyword analyzer on other.

Elasticsearch not working with 'not_analyzed' index

2 Answers