ElasticSearch NEST QueryParsingException bool query does not support filter

2
votes

I'm using NEST 2.3.2 to query ElasticSearch and need to create a conditional filter for FollowUpDateTime field based on optional boolean. If that boolean

  • is true - want to see all records with non-blank date,
  • is false - records without date,
  • is null - show all records.

Here's what I have in Nest: 

                    var containers = new QueryContainer[7];
                    //Size is based on hte number of search parameters.. 
                    var descriptor = new QueryContainerDescriptor<EsNoteModel>();

                    var filters = new QueryContainer[1];
                    var filterdescriptor = new QueryContainerDescriptor<EsNoteModel>();  

       /*...... some code for other parameters here .....*/

        if (hasFollowUpDate.HasValue)
                    {
                        if ((bool) hasFollowUpDate)
                        {
                            //If true shows only with a non-blank follow-up date
                            filters[0] = filterdescriptor.Exists(ff => ff.Field("FollowUpDateTime"));
                        }
                        else
                        {
                            //If false shows only notes with a blank follow-up date
                            filters[0] = filterdescriptor.Missing(ff => ff.Field("FollowUpDateTime"));
                        }
                    }

/*...... CALL to ES below .....*/    

                    var result = ElasticSearchClient.Search<EsNoteModel>(body => body
                        .From(offset - 1)
                        .Size(rows)
                        .Query(q => q
                            .Bool(b => b
                                  .Must(containers)
                                  .Filter(f => f.Bool(t => t.Must(filters)))
                                  )
                                )

                        .Sort(s => s
                            .Field(f => f
                                .Field(p => p.NoteDate)
                                .Order(SortOrder.Descending)))

                        );

The problem is that the query above generated the error. Please see below:

Invalid NEST response built from a unsuccessful low level call on POST: /_search?pretty=true

Audit trail of this API call:

  • BadResponse: Node: http://**********.es.amazonaws.com/ Took: 00:00:00.1381301

    OriginalException: System.Net.WebException: The remote server returned an error: (400) Bad Request. at

System.Net.HttpWebRequest.GetResponse() at Elasticsearch.Net.HttpConnection.Request[TReturn](RequestData requestData)

# Request:
{
  "from": 0,
  "size": 25,
  "sort": [
    {
      "NoteDate": {
        "order": "desc"
      }
    }
  ],
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "CustomerId": {
              "query": "2810855"
            }
          }
        }
      ],
      "filter": [
        {
          "bool": {
            "must": [
              {
                "exists": {
                  "field": "FollowUpDateTime"
                }
              }
            ]
          }
        }
      ]
    }
  }
}

Response below:

# Response:
{
  "error" : "SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[-t7SwBo_TjeH_98h_56ANw][customernote][0]: RemoteTransportException[[Neophyte][inet[/x.x.x.x:y]][indices:data/read/search[phase/query]]]; nested: SearchParseException[[customernote][0]: from[0],size[25],sort[<custom:\"NoteDate\": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@2d55231>!]: Parse Failure [Failed to parse source [{\r\n  \"from\": 0,\r\n  \"size\": 25,\r\n  \"sort\": [\r\n    {\r\n      \"NoteDate\": {\r\n        \"order\": \"desc\"\r\n      }\r\n    }\r\n  ],\r\n  \"query\": {\r\n    \"bool\": {\r\n      \"must\": [\r\n        {\r\n          \"match\": {\r\n            \"CustomerId\": {\r\n              \"query\": \"28_10855\"\r\n            }\r\n          }\r\n        }\r\n      ],\r\n      \"filter\": [\r\n        {\r\n          \"bool\": {\r\n            \"must\": [\r\n              {\r\n                \"exists\": {\r\n                  \"field\": \"FollowUpDateTime\"\r\n                }\r\n              }\r\n            ]\r\n          }\r\n        }\r\n      ]\r\n    }\r\n  }\r\n}]]]; nested: QueryParsingException[[customernote] bool query does not support [filter]]; }{[TOOugCq3ROqF9CGnHIay2g][customernote][1]: SearchParseException[[customernote][1]: from[0],size[25],sort[<custom:\"NoteDate\": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@562f97b5>!]: Parse Failure [Failed to parse source [{\r\n  \"from\": 0,\r\n  \"size\": 25,\r\n  \"sort\": [\r\n    {\r\n      \"NoteDate\": {\r\n        \"order\": \"desc\"\r\n      }\r\n    }\r\n  ],\r\n  \"query\": {\r\n    \"bool\": {\r\n      \"must\": [\r\n        {\r\n          \"match\": {\r\n            \"CustomerId\": {\r\n              \"query\": \"28_10855\"\r\n            }\r\n          }\r\n        }\r\n      ],\r\n      \"filter\": [\r\n        {\r\n          \"bool\": {\r\n            \"must\": [\r\n              {\r\n                \"exists\": {\r\n                  \"field\": \"FollowUpDateTime\"\r\n                }\r\n              }\r\n            ]\r\n          }\r\n        }\r\n      ]\r\n    }\r\n  }\r\n}]]]; nested: QueryParsingException[[customernote] bool query does not support [filter]]; }{[TOOugCq3ROqF9CGnHIay2g][customernote][2]: SearchParseException[[customernote][2]: from[0],size[25],sort[<custom:\"NoteDate\": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@394cf46e>!]: Parse Failure [Failed to parse source [{\r\n  \"from\": 0,\r\n  \"size\": 25,\r\n  \"sort\": [\r\n    {\r\n      \"NoteDate\": {\r\n        \"order\": \"desc\"\r\n      }\r\n    }\r\n  ],\r\n  \"query\": {\r\n    \"bool\": {\r\n      \"must\": [\r\n        {\r\n          \"match\": {\r\n            \"CustomerId\": {\r\n              \"query\": \"28_10855\"\r\n            }\r\n          }\r\n        }\r\n      ],\r\n      \"filter\": [\r\n        {\r\n          \"bool\": {\r\n            \"must\": [\r\n              {\r\n                \"exists\": {\r\n                  \"field\": \"FollowUpDateTime\"\r\n                }\r\n              }\r\n            ]\r\n          }\r\n        }\r\n      ]\r\n    }\r\n  }\r\n}]]]; nested: QueryParsingException[[customernote] bool query does not support [filter]]; }{[-t7SwBo_TjeH_98h_56ANw][customernote][3]: RemoteTransportException[[Neophyte][inet[/x.x.x.x:y]][indices:data/read/search[phase/query]]]; nested: SearchParseException[[customernote][3]: from[0],size[25],sort[<custom:\"NoteDate\": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@46d07ba8>!]: Parse Failure [Failed to parse source [{\r\n  \"from\": 0,\r\n  \"size\": 25,\r\n  \"sort\": [\r\n    {\r\n      \"NoteDate\": {\r\n        \"order\": \"desc\"\r\n      }\r\n    }\r\n  ],\r\n  \"query\": {\r\n    \"bool\": {\r\n      \"must\": [\r\n        {\r\n          \"match\": {\r\n            \"CustomerId\": {\r\n              \"query\": \"28_10855\"\r\n            }\r\n          }\r\n        }\r\n      ],\r\n      \"filter\": [\r\n        {\r\n          \"bool\": {\r\n            \"must\": [\r\n              {\r\n                \"exists\": {\r\n                  \"field\": \"FollowUpDateTime\"\r\n                }\r\n              }\r\n            ]\r\n          }\r\n        }\r\n      ]\r\n    }\r\n  }\r\n}]]]; nested: QueryParsingException[[customernote] bool query does not support [filter]]; }{[TOOugCq3ROqF9CGnHIay2g][customernote][4]: SearchParseException[[customernote][4]: from[0],size[25],sort[<custom:\"NoteDate\": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@74fb3710>!]: Parse Failure [Failed to parse source [{\r\n  \"from\": 0,\r\n  \"size\": 25,\r\n  \"sort\": [\r\n    {\r\n      \"NoteDate\": {\r\n        \"order\": \"desc\"\r\n      }\r\n    }\r\n  ],\r\n  \"query\": {\r\n    \"bool\": {\r\n      \"must\": [\r\n        {\r\n          \"match\": {\r\n            \"CustomerId\": {\r\n              \"query\": \"28_10855\"\r\n            }\r\n          }\r\n        }\r\n      ],\r\n      \"filter\": [\r\n        {\r\n          \"bool\": {\r\n            \"must\": [\r\n              {\r\n                \"exists\": {\r\n                  \"field\": \"FollowUpDateTime\"\r\n                }\r\n              }\r\n            ]\r\n          }\r\n        }\r\n      ]\r\n    }\r\n  }\r\n}]]]; nested: QueryParsingException[[customernote] bool query does not support [filter]]; }]",
  "status" : 400
}

As you see error reports "nested: QueryParsingException[[customernote] bool query does not support [filter]];" Any ideas what I'm doing wrong in this example? Is this a proper use of filter clause?

**** UPDATE

changed query 

 var result = ElasticSearchClient.Search<EsNoteModel>(body => body
                .From(offset - 1)
                .Size(rows)
                .Query(q => q
                    .Bool(b => b
                          .Must(containers)
                          .Filter(filters)
                          )
                        )

                .Sort(s => s
                    .Field(f => f
                        .Field(p => p.NoteDate)
                        .Order(SortOrder.Descending)))

                );

request to 

# Request:
{
  "from": 0,
  "size": 25,
  "sort": [
    {
      "NoteDate": {
        "order": "desc"
      }
    }
  ],
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "CustomerId": {
              "query": "2810855"
            }
          }
        }
      ],
      "filter": [
        {
          "exists": {
            "field": "FollowUpDateTime"
          }
        }
      ]
    }
  }
}

getting response error:

# Response:
{
  "error" : "SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[TOOugCq3ROqF9CGnHIay2g][customernote][0]: SearchParseException[[customernote][0]: from[0],size[25],sort[<custom:\"NoteDate\": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@4839a60c>!]: Parse Failure [Failed to parse source [{\r\n  \"from\": 0,\r\n  \"size\": 25,\r\n  \"sort\": [\r\n    {\r\n      \"NoteDate\": {\r\n        \"order\": \"desc\"\r\n      }\r\n    }\r\n  ],\r\n  \"query\": {\r\n    \"bool\": {\r\n      \"must\": [\r\n        {\r\n          \"match\": {\r\n            \"CustomerId\": {\r\n              \"query\": \"2810855\"\r\n            }\r\n          }\r\n        }\r\n      ],\r\n      \"filter\": [\r\n        {\r\n          \"exists\": {\r\n            \"field\": \"FollowUpDateTime\"\r\n          }\r\n        }\r\n      ]\r\n    }\r\n  }\r\n}]]]; nested: QueryParsingException[[customernote] bool query does not support [filter]]; }{[-t7SwBo_TjeH_98h_56ANw][customernote][1]: RemoteTransportException[[Neophyte][inet[/x.x.x.x:y]][indices:data/read/search[phase/query]]]; nested: SearchParseException[[customernote][1]: from[0],size[25],sort[<custom:\"NoteDate\": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@21d845ae>!]: Parse Failure [Failed to parse source [{\r\n  \"from\": 0,\r\n  \"size\": 25,\r\n  \"sort\": [\r\n    {\r\n      \"NoteDate\": {\r\n        \"order\": \"desc\"\r\n      }\r\n    }\r\n  ],\r\n  \"query\": {\r\n    \"bool\": {\r\n      \"must\": [\r\n        {\r\n          \"match\": {\r\n            \"CustomerId\": {\r\n              \"query\": \"2810855\"\r\n            }\r\n          }\r\n        }\r\n      ],\r\n      \"filter\": [\r\n        {\r\n          \"exists\": {\r\n            \"field\": \"FollowUpDateTime\"\r\n          }\r\n        }\r\n      ]\r\n    }\r\n  }\r\n}]]]; nested: QueryParsingException[[customernote] bool query does not support [filter]]; }{[-t7SwBo_TjeH_98h_56ANw][customernote][2]: RemoteTransportException[[Neophyte][inet[/x.x.x.x:y]][indices:data/read/search[phase/query]]]; nested: SearchParseException[[customernote][2]: from[0],size[25],sort[<custom:\"NoteDate\": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@6124d214>!]: Parse Failure [Failed to parse source [{\r\n  \"from\": 0,\r\n  \"size\": 25,\r\n  \"sort\": [\r\n    {\r\n      \"NoteDate\": {\r\n        \"order\": \"desc\"\r\n      }\r\n    }\r\n  ],\r\n  \"query\": {\r\n    \"bool\": {\r\n      \"must\": [\r\n        {\r\n          \"match\": {\r\n            \"CustomerId\": {\r\n              \"query\": \"2810855\"\r\n            }\r\n          }\r\n        }\r\n      ],\r\n      \"filter\": [\r\n        {\r\n          \"exists\": {\r\n            \"field\": \"FollowUpDateTime\"\r\n          }\r\n        }\r\n      ]\r\n    }\r\n  }\r\n}]]]; nested: QueryParsingException[[customernote] bool query does not support [filter]]; }{[TOOugCq3ROqF9CGnHIay2g][customernote][3]: SearchParseException[[customernote][3]: from[0],size[25],sort[<custom:\"NoteDate\": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@f884425>!]: Parse Failure [Failed to parse source [{\r\n  \"from\": 0,\r\n  \"size\": 25,\r\n  \"sort\": [\r\n    {\r\n      \"NoteDate\": {\r\n        \"order\": \"desc\"\r\n      }\r\n    }\r\n  ],\r\n  \"query\": {\r\n    \"bool\": {\r\n      \"must\": [\r\n        {\r\n          \"match\": {\r\n            \"CustomerId\": {\r\n              \"query\": \"2810855\"\r\n            }\r\n          }\r\n        }\r\n      ],\r\n      \"filter\": [\r\n        {\r\n          \"exists\": {\r\n            \"field\": \"FollowUpDateTime\"\r\n          }\r\n        }\r\n      ]\r\n    }\r\n  }\r\n}]]]; nested: QueryParsingException[[customernote] bool query does not support [filter]]; }{[-t7SwBo_TjeH_98h_56ANw][customernote][4]: RemoteTransportException[[Neophyte][inet[/x.x.x.x:y]][indices:data/read/search[phase/query]]]; nested: SearchParseException[[customernote][4]: from[0],size[25],sort[<custom:\"NoteDate\": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@77228f64>!]: Parse Failure [Failed to parse source [{\r\n  \"from\": 0,\r\n  \"size\": 25,\r\n  \"sort\": [\r\n    {\r\n      \"NoteDate\": {\r\n        \"order\": \"desc\"\r\n      }\r\n    }\r\n  ],\r\n  \"query\": {\r\n    \"bool\": {\r\n      \"must\": [\r\n        {\r\n          \"match\": {\r\n            \"CustomerId\": {\r\n              \"query\": \"2810855\"\r\n            }\r\n          }\r\n        }\r\n      ],\r\n      \"filter\": [\r\n        {\r\n          \"exists\": {\r\n            \"field\": \"FollowUpDateTime\"\r\n          }\r\n        }\r\n      ]\r\n    }\r\n  }\r\n}]]]; nested: QueryParsingException[[customernote] bool query does not support [filter]]; }]",
  "status" : 400
}

Basically same error

nested: QueryParsingException[[customernote] bool query does not support [filter]];

2
amazon-web-serviceselasticsearchnest
Bool queries can only have a filter from elasticsearch 2 onwards. – Frederick Cheung
using Elasticsearch.Net.2.3.2 – Nat
That's the client you're using but the server is 1.5.2 (since that is the only version provided by the aws elasticsearch service, which you appear to be using) – Frederick Cheung
Frederick - you deserve a star! thank you – Nat
The major versions of NEST are compatible with the major versions of Elasticsearch so for Elasticsearch 1.5.2, you should use the latest NEST 1.x, which at this time is 1.8.2. Check out the compatibility matrix on github - github.com/elastic/elasticsearch-net – Russ Cam

2 Answers

1
votes

Apparently we have in out .NET 4.0 project the following references to

  1. Elasticsearch.Net - official low level elasticsearch client packages\Elasticsearch.Net.2.3.2\lib\net45\Elasticsearch.Net.dll
  2. NEST - official high level elasticsearch client packages\NEST.2.3.2\lib\net45\Nest.dll

interesting fact they still use deprecated queries. as mentioned by Frederick here

the client you're using but the server is 1.5.2 (since that is the only version provided by the aws elasticsearch service, which you appear to be using)

see below - it works:

            var query = new QueryContainer[8];//Size is based on hte number of search parameters
            var descriptor = new QueryContainerDescriptor<EsNoteModel>();

/*........... skipping here some code for other parameters ..........*/

 if (hasFollowUpDate.HasValue)
            {
                if ((bool) hasFollowUpDate)
                {
                    //If true shows only with a non-blank follow-up date
                    query[7] = descriptor.Filtered(p => p.Filter(f => f.Exists(r => r.Field(u => u.FollowUpDateTime))));
                }
                else
                {
                    ////If false shows only notes with a blank follow-up date
                    query[7] = descriptor.Filtered(p => p.Filter(f => f.Missing(r => r.Field(u => u.FollowUpDateTime))));
                }
            }

            var result = ElasticSearchClient.Search<EsNoteModel>(body => body
                .From(offset - 1)
                .Size(rows)
                .Query(q => q
                    .Bool(b => b
                          .Must(query)
                          )
                        )

                .Sort(s => s
                    .Field(f => f
                        .Field(p => p.NoteDate)
                        .Order(SortOrder.Descending)))

                );

So simply replacing word BOOL to Filtered - did the trick

0
votes

You can do this dynamically using a QueryContainer or FilterContainer. Here is a very similar one from our solution:

var query = new List<QueryContainer>
{
    Query<AlertForReporting>.Range(r => r.OnField(f => f.AlertDate)
                                          .Greater(start.ToEPCISFormat())
                                          .Lower(end.ToEPCISFormat())
        )
};


var result = ElasticClient.Search<AlertForReporting>(s =>
                        s.Size(0)
                            .Query(q =>
                                q.Bool(b =>
                                        b.Must(query.ToArray())
                                    )
                            ));