An application server is sending syslog messages to a Logstash instance. These syslog messages does contain beside other information also Java stack traces. The problem is that the stack trace is divided into multiple syslog messages. I need to aggregate all these events to just one, which does contain the whole stack trace.
The syslog messages which are sent to the Logstash instance line based look like the following:
Aug 2 16:01:51 hostname app-name: app.fonctionnel.fluxfi.outils.exception.FluxfiException: Erreur lors de la réception du guide prestation
Aug 2 16:01:51 hostname app-name: at app.fonctionnel.fluxfi.proxy.Tech_WWFluxFiJdoSin.RecevoirGuidePrest(Tech_WWFluxFiJdoSin.java:89)
Aug 2 16:01:51 hostname app-name: at app.fonctionnel.fluxfi.world.Inter_WWFLUXFI_General.FF_RecevoirGuidePrest(Inter_WWFLUXFI_General.java:1103)
Aug 2 16:01:51 hostname app-name: at app.fonctionnel.prest.interdomaines.ffi.interfacesappprivees.InterfaceappPrivee_Ffi.PreV2_EnvoyerGuidePrest(InterfaceappPrivee_Ffi.java:70)
Aug 2 16:01:51 hostname app-name: at app.fonctionnel.prest.interdomaines.ffi.interfacesappprivees.InterfaceappPrivee_Ffi.PreV2_EnvoyerGuidePrest(InterfaceappPrivee_Ffi.java:156)
Aug 2 16:01:51 hostname app-name: at app.fonctionnel.prest.interdomaines.ffi.services.InterfaceFfiImpl.envoyerGuidePrest(InterfaceFfiImpl.java:14)
Aug 2 16:01:51 hostname app-name: at app.fonctionnel.prest.services.MouvementsManagerImpl.transfertMvts(MouvementsManagerImpl.java:164)
Aug 2 16:01:51 hostname app-name: at app.fonctionnel.prest.services.TrtPrestations.TransfererDecompte(TrtPrestations.java:1692)
Aug 2 16:01:51 hostname app-name: at app.fonctionnel.prest.services.TrtPrestations.TransfererDecompte(TrtPrestations.java:1655)
Aug 2 16:01:51 hostname app-name: at app.fonctionnel.prest.facades.FcdMvtPrestation.TransfererDecompte(FcdMvtPrestation.java:690)
Aug 2 16:01:51 hostname app-name: at app.fonctionnel.prest.proxy.PreV2_Macro.ValiderDecompte(PreV2_Macro.java:3192)
Aug 2 16:01:51 hostname app-name: at app.fonctionnel.prest.servlet.Pre_Prestation$Pre_Prestation_Macros.traiterMacro(Pre_Prestation.java:218)
Aug 2 16:01:51 hostname app-name: at app.fonctionnel.prest.servlet.Pre_Prestation.traiterMacro(Pre_Prestation.java:130)
Aug 2 16:01:51 hostname app-name: at app.serveur.fwgraphic.thinclient.Tech_ServletBase.traiterAction(Tech_ServletBase.java:820)
Aug 2 16:01:51 hostname app-name: at app.serveur.fwgraphic.thinclient.Tech_ServletBase.traiterRequete(Tech_ServletBase.java:691)
Aug 2 16:01:51 hostname app-name: at app.serveur.fwgraphic.thinclient.Tech_ServletBase.handleNormal(Tech_ServletBase.java:408)
Aug 2 16:01:51 hostname app-name: at app.serveur.fwgraphic.thinclient.Tech_ServletBase.processRequest(Tech_ServletBase.java:326)
Aug 2 16:01:51 hostname app-name: at nsi.serveur.fwgraphic.thinclient.Tech_ServletBase.service(Tech_ServletBase.java:231)
Aug 2 16:01:51 hostname app-name: at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
Aug 2 16:01:51 hostname app-name: at ch.generali.fwk.web.filter.InvokerLoadListener$WrapperServlet.service(InvokerLoadListener.java:216)
Aug 2 16:01:51 hostname app-name: at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
Aug 2 16:01:51 hostname app-name: at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
Aug 2 16:01:51 hostname app-name: at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
Aug 2 16:01:51 hostname app-name: at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
Current Logstash configuration:
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{SYSLOGHOST:hostname} %{DATA:application}(?:\[%{POSINT:pid}\])?: %{GREEDYDATA:raw_message}" }
}
date {
match => [ "timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]
timezone => "Europe/Zurich"
}
if [application] == "app-name" {
aggregate {
task_id => "%{timestamp}"
code => "
map['stack_trace_entry'] = event.get('raw_message')
"
push_previous_map_as_event => true
timeout => 3
}
}
mutate {
remove_field => ["host", "timestamp"]
}
if [hostname] == "app-name.example.com" {
json {
source => "raw_message"
}
mutate {
add_field => {"[@metadata][es_index]" => "app-name-devl-logs-%{+YYYY.MM.dd}"}
add_field => {"[@metadata][document_id]" => "%{[req_id]}" }
remove_field => ["raw_message"]
}
} else {
fingerprint {
source => "message"
target => "[@metadata][document_id]"
method => "MURMUR3"
}
mutate {
add_field => {"[@metadata][es_index]" => "app-name-devl-logs-%{+YYYY.MM.dd}"}
remove_field => ["message"]
}
}
}
}
I tried the following options:
- Aggregate filter
- Multiline
However none of the above approaches worked out for me.
Does somebody have a snippet or a solution for this problem?
