Querying large amounts of terms without expanding maxClauseCount

2
votes

In a data flow of mine, I am trying to retrieve a subset of documents from a previous terms aggregation, but hitting the maxClauseCount limit within my ES cluster. The follow up query is along these lines:

GET dataset/_search
{
  "size": 2000,
  "query": {
    "bool": {
      "must": [
        (a filter or two)...,
        {
          "terms":{ 
              "otherid":[
                  "789e18f2-bacb-4e38-9800-bf8e4c65c206",
                  "8e6967aa-5b98-483e-b50f-c681c7396a6a",
                  ...
              ]
           }
         }
      ]}
   }
}

In my research I've come across a lookup - which sadly we can't use - as well as the ids query.

https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-terms-query.html https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-ids-query.html

From experimentation, it appears that the ids query doesn't share the limit the terms query has (potentially it's not converted into terms clauses). Do any of you know if there's a good way to achieve similar functionality to the ids query without using the ids fields.

My version of ES is 5.0.

Thanks!

1
elasticsearch

1 Answers

3
votes

instead of using terms use the Terms filter it will solve the issue

OR index.query.bool.max_clause_count: increase to higher value(*Not Recommended)

http://george-stathis.com/2013/10/18/setting-the-booleanquery-maxclausecount-in-elasticsearch/