Logstash log parsing

Question

I've got this log message:

Jul 23 09:24:16 mmr mmr-core[5147]:  Aweg3AOMTs_1563866656876839.mt GetProvider_v4

which I ship to the elasticsearch.

I'd like to parse it in logstash filter and make id field from Aweg3AOMTs_1563866656876839.mt except I don't want to involve either letter or other characters and have in the id only the number!

I've done so far:

%{SYSLOGTIMESTAMP:logtimestamp} %{HOSTNAME:hostname} %{DATA:type} %{USERNAME:id} %{GREEDYDATA:rest}

  "logtimestamp": "Jul 23 09:24:16",
  "hostname": "mmr",
  "id": "Aweg3AOMTs_1563866656876839.mt",
  "type": "mmr-core[5147]:"

How can I skip the letters and characters in the id field?

Thank you for any help!

What are the possible formats that this id can be in? Can you be sure that all the numbers will be in a row without any other characters between them? — mihomir
@mihomir it's only format the id can be in :)) All numbers are in a row without characters between ! — vladpoverin

little_pinecone little_pinecone · Accepted Answer · 2020-05-19T09:28:56

As far as I understand, you want just the 1563866656876839 in your id field? You could divide the Aweg3AOMTs_1563866656876839.mt string into three fields: id_prefix, id and id_suffix and then remove the parts that you don't need.

Pattern to try in the Grok Debugger:

%{SYSLOGTIMESTAMP:logtimestamp}\s%{HOSTNAME:hostname}\s%{DATA:type}\s%{USERNAME:id_prefix}_%{NUMBER:id}\.%{GREEDYDATA:id_suffix}\s%{GREEDYDATA:rest}

Logstash config:

# logstash.conf
…
filter {
    grok {
        match => {
            "message" => "%{SYSLOGTIMESTAMP:logtimestamp}\s%{HOSTNAME:hostname}\s%{DATA:type}\s%{USERNAME:id_prefix}_%{NUMBER:id}\.%{GREEDYDATA:id_suffix}\s%{GREEDYDATA:rest}"
        }
    }
    mutate {
        remove_field => ["id_prefix", "id_suffix"]
    }
}
…

Logstash log parsing

1 Answers