Logstash Add field from grok filter

Question

Is it possible to match a message to a new field in logstash using grok and mutate?

Example log:

"<30>Dec 19 11:37:56 7f87c507df2a[20103]: [INFO] 2018-12-19 16:37:56 _internal (MainThread): 192.168.0.6 - - [19/Dec/2018 16:37:56] \"\u001b[37mGET / HTTP/1.1\u001b[0m\" 200 -\r"

I am trying to create a new key value where I match container_id to 7f87c507df2a.

filter {
  grok {
    match => [ "message", "%{SYSLOG5424PRI}%{NONNEGINT:ver} +(?:%{TIMESTAMP_ISO8601:ts}|-) +(?:%{HOSTNAME:service}|-) +(?:%{NOTSPACE:containerName}|-) +(?:%{NOTSPACE:proc}|-) +(?:%{WORD:msgid}|-) +(?:%{SYSLOG5424SD:sd}|-|) +%{GREEDYDATA:msg}" ]
  }
  mutate {
    add_field => { "container_id" => "%{containerName}"}
  }
}

The resulting logfile renders this, where the value of containerName isn't being referenced from grok, it is just a string literal:

"container_id": "%{containerName}"

I am trying to have the conf create:

"container_id": "7f87c507df2a"

Obviously the value of containerName isn't being linked from grok. Is what I want to do even possible?

I am not sure if I unterstand the question, but if so a simple filter { mutate { copy => { "containerName" => "container_id" } } } should do the job — Kali
anther thing you can try is adding add_field => { "container_id" => "%{containerName}"} directly into the grok part - but if "containerName" is not matched in the grok pattern the key/value pair is not added to the result. — Kali
@Quali neither have worked, is there an easier way to test that my grok matching is working properly? There seems to be no errors in the logs of logstash however. I updated the post with an example log file and what I am trying to do — Baily

Baily Baily · Accepted Answer · 2018-12-19T17:56:33

As explained in the comments, my grok pattern was incorrect. For anyone that may wander towards this post that needs help with grok go here to make building your pattern less time consuming.

Here was the working snapshot:

filter {
  grok {
    match => [ "message", "\A%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP}%{SPACE}%{BASE16NUM:docker_id}%{SYSLOG5424SD}%{GREEDYDATA:python_log_message}" ]
    add_field => { "container_id" => "%{docker_id}" }    
  }  
}

Logstash Add field from grok filter

1 Answers