grok parser failure - for django logs

Question

This is 1 of my log entries,

INFO 2017-05-16 17:24:11,690 views 14463 139643033982720 https://play.google.com/store/apps/details?id=com.VoDrive&referrer=referral_code%3DP5E

This is my pattern ,

DJANGOTIMESTAMP %{YEAR}-%{MONTHNUM}-%{MONTHDAY}%{SPACE}%{HOUR}:%{MINUTE}:%{SECOND}

This is my logstash conf file,

input {
  beats {
    port => "5043"
 }
}
filter {
  if [type] in ["django"] {
    grok {
      patterns_dir => ["/opt/logstash/patterns"]
      match => [ "message" , "%{LOGLEVEL:level}%{SPACE}%{DJANGOTIMESTAMP:timestamp},%{INT:pid}%{SPACE}%{WORD:origin}%{SPACE}%{INT:uid}%{SPACE}%{INT:django-id}%{SPACE}%{GREEDYDATA:action}" ]
    }
 }
 }
 output {
   elasticsearch {
    hosts => [ "localhost:9200" ]
    index => "%{type}_indexer"
 }
 }

IN elasticsearch output, the fields are not made,

luvpreet@DHARI-Inspiron-3542:/usr/bin$ curl -XGET 'localhost:9200/django_indexer/_search?pretty=true&q=*:*'
      {
    "_index" : "django_indexer",
    "_type" : "django",
    "_id" : "AVwu8tE7j-Kh6vl1kUdf",
    "_score" : 1.0,
    "_source" : {
      "@timestamp" : "2017-05-22T06:55:52.819Z",
      "offset" : 144,
      "@version" : "1",
      "beat" : {
        "hostname" : "DHARI-Inspiron-3542",
        "name" : "DHARI-Inspiron-3542",
        "version" : "5.4.0"
      },
      "input_type" : "log",
      "host" : "DHARI-Inspiron-3542",
      "source" : "/var/log/django/a.log",
      "message" : "INFO 2017-05-16 06:33:08,673 views 40152 139731056719616 https://play.google.com/store/apps/details?id=com.VoDrive&referrer=referral_code%3DP5E",
      "type" : "django",
      "tags" : [
        "beats_input_codec_plain_applied"
      ]
    }

It is not saying that parser has failed, but why are the fields not being made ? What am I lacking ?

berrytchaks berrytchaks · Accepted Answer · 2017-05-19T08:55:58

Try with this grok pattern:

%{LOGLEVEL:loglevel}%{SPACE}%{TIMESTAMP_ISO8601:timestamp},%{INT:pid}%{SPACE}%{WORD:origin}%{SPACE}%{INT:id}%{SPACE}%{INT:number}%{SPACE}%{URI:action}

Input

INFO 2017-05-16 17:24:11,690 views 14463 139643033982720 https://play.google.com/store/apps/details?id=com.VoDrive&referrer=referral_code%3DP5E

Output

number      139643033982720
timestamp   2017-05-16·17:24:11
id          14463
port    
pid         690
origin      views
action      https://play.google.com/store/apps/details?id=com.VoDrive&referrer=referral_code%3DP5E
loglevel    INFO

You can then remove the field port with a mutate in you filter plugin

mutate {
    remove_field => ["port"]
}

UPDATE

Ok, I tried your configuration on with my logstash. This is what I did:

1- Configure filebeat:

filebeat.prospectors:

- paths:
    - /etc/filebeat/FilebeatInputTest.txt
  document_type: django

output.logstash:
  hosts: ["127.0.0.1:5044"]

2- Configure logstash

input {
  beats {
    port => "5044"
 }
}
filter {
  if [type] == "django" {
    grok {
      match => [ "message" , "%{LOGLEVEL:loglevel}%{SPACE}%{TIMESTAMP_ISO8601:timestamp},%{INT:pid}%{SPACE}%{WORD:origin}%{SPACE}%{INT:id}%{SPACE}%{INT:number}%{SPACE}%{GREEDYDATA:action}" ]
    }
    mutate {
        remove_field => ["@timestamp", "beat","input_type","offset","source","@version","host","tags","message"]
    }
 }
}
output {
   elasticsearch {
    hosts => [ "xx.xx.xx.xx:9200" ]
    index => "%{type}_indexer"
    user => "xxxx"
    password => "xxxx"
  }
}

You can remove user and password if your elasticsearch is not secured.

Input (content of /etc/filebeat/FilebeatInputTest.txt)

INFO 2017-05-16 17:24:11,690 views 14463 139643033982720 https://play.google.com/store/apps/details?id=com.VoDrive&referrer=referral_code%3DP5E

Output (In elasticsearch)

{
    "_index" : "django_indexer",
    "_type" : "django",
    "_id" : "AVwhFe30JYGYNG_7C7YI",
    "_score" : 1.0,
    "_source" : {
        "origin" : "views",
        "pid" : "690",
        "type" : "django",
        "number" : "139643033982720",
        "loglevel" : "INFO",
        "action" : "https://play.google.com/store/apps/details?id=com.VoDrive&referrer=referral_code%3DP5E",
        "id" : "14463",
        "timestamp" : "2017-05-16 17:24:11"
    }
}

Hope this helps.

grok parser failure - for django logs

1 Answers