Elasticsearch 5.X Percolate: How to autogenerate copy_to fields?

Question

In ES 2.3.3, many queries in the system I'm working on use the _all field. Sometimes these are registered to a percolate index, and when running percolator on the doc, _all is generated automatically.

In converting to ES 5.X _all is being deprecated and so _all has been replaced with a copy_to field that contains the components that we actually care about, and it works great for those searches.

Registering the same query to a percolate index with the same document mapping including copy_to fields works fine. Sending a percolate query with the document never results in a hit for a copy_to field however.

Manually building the copy_to field via simple string concatenation seems to work, it's just that I'd expect to be able to Query -> DocIndex and get the same result as Doc -> PercolateQuery... So I'm just looking for a way to have ES generate the copy_to fields automatically on a document being percolated.

Jon Dewitt Jon Dewitt · Accepted Answer · 2018-04-10T04:13:21

Ended up there was nothing wrong with ES of course, posting here in case it helps someone else. Figured it out while attempting to generate a simpler example to post here with details... Basically the issue came down to the fact that attempting to percolate a document of a type that doesn't exist in the percolate index doesn't give any errors back, but seems to apply all percolate queries without applying any mappings which was just confusing as it worked for simple test cases, but not complex ones. Here's an example:

From the copy_to docs, generate an index with a copy_to mapping. See that a query to the copy_to field works.

PUT my_index
{
  "mappings": {
    "my_type": {
      "properties": {
        "first_name": {
          "type": "text",
          "copy_to": "full_name" 
        },
        "last_name": {
          "type": "text",
          "copy_to": "full_name" 
        },
        "full_name": {
          "type": "text"
        }
      }
    }
  }
}

PUT my_index/my_type/1
{
  "first_name": "John",
  "last_name": "Smith"
}

GET my_index/_search
{
  "query": {
    "match": {
      "full_name": { 
        "query": "John Smith",
        "operator": "and"
      }
    }
  }
}

Create a percolate index with the same type

PUT /my_percolate_index
{
  "mappings": {
    "my_type": {
      "properties": {
        "first_name": {
          "type": "text",
          "copy_to": "full_name" 
        },
        "last_name": {
          "type": "text",
          "copy_to": "full_name" 
        },
        "full_name": {
          "type": "text"
        }
      }
    },
    "queries": {
      "properties": {
        "query": {
          "type": "percolator"
        }
      }
    }
  }
}

Create a percolate query that matches our other percolate query on the copy_to field, and a second query that just queries on a basic unmodified field

PUT /my_percolate_index/queries/1?refresh
{
  "query": {
    "match": {
      "full_name": { 
        "query": "John Smith",
        "operator": "and"
      }
    }
  }
}
PUT /my_percolate_index/queries/2?refresh
{
  "query": {
    "match": {
      "first_name": { 
        "query": "John"
      }
    }
  }
}

Search, but with the wrong type... there will be a hit on the basic field (first_name: John) even though no document mappings match the request

GET /my_percolate_index/_search
{
  "query" : {
    "percolate" : {
      "field" : "query",
      "document_type" : "non_type",
      "document" : {
        "first_name": "John",
        "last_name": "Smith"
      }
    }
  }
}
{"took":7,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":1,"max_score":0.2876821,"hits":[{"_index":"my_percolate_index","_type":"queries","_id":"2","_score":0.2876821,"_source":{
  "query": {
    "match": {
      "first_name": { 
        "query": "John"
      }
    }
  }
}}]}}

Send in the correct document_type and see both matches as expected

GET /my_percolate_index/_search
{
  "query" : {
    "percolate" : {
      "field" : "query",
      "document_type" : "my_type",
      "document" : {
        "first_name": "John",
        "last_name": "Smith"
      }
    }
  }
}
{"took":7,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":2,"max_score":0.51623213,"hits":[{"_index":"my_percolate_index","_type":"queries","_id":"1","_score":0.51623213,"_source":{
  "query": {
    "match": {
      "full_name": { 
        "query": "John Smith",
        "operator": "and"
      }
    }
  }
}},{"_index":"my_percolate_index","_type":"queries","_id":"2","_score":0.2876821,"_source":{
  "query": {
    "match": {
      "first_name": { 
        "query": "John"
      }
    }
  }
}}]}}

Elasticsearch 5.X Percolate: How to autogenerate copy_to fields?

1 Answers