How to get Elasticsearch boolean match working for multiple fields

Question

I need some expert guidance on trying to get a bool match working. I'd like the query to only return a successful search result if both 'message' matches 'Failed password for', and 'path' matches '/var/log/secure'.

This is my query:

curl -s -XGET 'http://localhost:9200/logstash-2015.05.07/syslog/_search?pretty=true' -d '{
    "filter" : { "range" : { "@timestamp" : { "gte" : "now-1h" } } },
    "query" : {
        "bool" : {
            "must" : [
                {  "match_phrase" : { "message" : "Failed password for" } },
                {  "match_phrase" : { "path"    : "/var/log/secure"     } }
            ]
        }
    }
} '

Here is the start of the output from the search:

{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 46,
    "max_score" : 13.308596,
    "hits" : [ {
      "_index" : "logstash-2015.05.07",
      "_type" : "syslog",
      "_id" : "AU0wzLEqqCKq_IPSp_8k",
      "_score" : 13.308596,
      "_source":{"message":"May  7 16:53:50 s_local@logstash-02 sshd[17970]: Failed password for fred from 172.28.111.200 port 43487 ssh2","@version":"1","@timestamp":"2015-05-07T16:53:50.554-07:00","type":"syslog","host":"logstash-02","path":"/var/log/secure"}
    }, ...

The problem is if I change '/var/log/secure' to just 'var' say, and run the query, I still get a result, just with a lower score. I understood the bool...must construct meant both match terms here would need to be successful. What I'm after is no result if 'path' doesn't exactly match '/var/log/secure'...

{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 46,
    "max_score" : 10.354593,
    "hits" : [ {
      "_index" : "logstash-2015.05.07",
      "_type" : "syslog",
      "_id" : "AU0wzLEqqCKq_IPSp_8k",
      "_score" : 10.354593,
      "_source":{"message":"May  7 16:53:50 s_local@logstash-02 sshd[17970]: Failed password for fred from 172.28.111.200 port 43487 ssh2","@version":"1","@timestamp":"2015-05-07T16:53:50.554-07:00","type":"syslog","host":"logstash-02","path":"/var/log/secure"}
    },...

I checked the mappings for these fields to check that they are not analyzed :

curl -X GET 'http://localhost:9200/logstash-2015.05.07/_mapping?pretty=true'

I think these fields are non analyzed and so I believe the search will not be analyzed too (based on some training documentation I read recently from elasticsearch). Here is a snippet of the output _mapping for this index below.

      ....
      "message" : {
        "type" : "string",
        "norms" : {
          "enabled" : false
        },
        "fields" : {
          "raw" : {
            "type" : "string",
            "index" : "not_analyzed",
            "ignore_above" : 256
          }
        }
      },
      "path" : {
        "type" : "string",
        "norms" : {
          "enabled" : false
        },
        "fields" : {
          "raw" : {
            "type" : "string",
            "index" : "not_analyzed",
            "ignore_above" : 256
          }
        }
      },
      ....

Where am I going wrong, or what am I misunderstanding here?

keety keety · Accepted Answer · 2015-05-08T01:35:25

As mentioned in the OP you would need to use the "not_analyzed" view of the fields but as per the OP mapping the non-analyzed version of the field is message.raw, path.raw Example:

{
    "filter" : { "range" : { "@timestamp" : { "gte" : "now-1h" } } },
    "query" : {
        "bool" : {
            "must" : [
                {  "match_phrase" : { "message.raw" : "Failed password for" } },
                {  "match_phrase" : { "path.raw"    : "/var/log/secure"     } }
            ]
        }
    }
}

.The link alongside gives more insight to multi-fields

.To expand further

The mapping in the OP for path is as follows:

"path" : {
        "type" : "string",
        "norms" : {
          "enabled" : false
        },
        "fields" : {
          "raw" : {
            "type" : "string",
            "index" : "not_analyzed",
            "ignore_above" : 256
          }
        }
      }

This specifies that the path field uses the default analyzer and field.raw is not analyzed.

If you want to set the path field to be not analyzed instead of raw it would be something on these lines:

"path" : {
            "type" : "string",
            "index" : "not_analyzed",
            "norms" : {
              "enabled" : false
            },
            "fields" : {
              "raw" : {
                "type" : "string",
                "index" : <whatever analyzer you want>,
                "ignore_above" : 256
              }
            }
          }

How to get Elasticsearch boolean match working for multiple fields

1 Answers