Is it possible to create custom fields in a Kibana dashboard?

0
votes

I am using a Java micro-service architecture in my application and generating separate log files for each micro-service.

I am using ELK stack approach to visualize the logs in Kibana, but the problem is whatever the fields that I'm getting from Elastic Search that are related to server logs fields. some example fields are @timestamp,@version,@path,@version.keyword,@host.

i want to customize this fields by adding some fields like customerId,txn-Id,mobile no so that we can analyze the data easily.

I'm using org.apache.logging.log4j2 to write the logs. Can I set above fields (customerId,txn-Id,mobile) to log files? And then Elastic will store these fields with the above default fields and then these custom fields should available in a Kibana dashboard. Is this possible?

enter image description here

2
javaloggingmicroserviceskibanaelastic-stack

2 Answers

0
votes

It's definitely possible to do that. I've not done it with the log4j2 stack (I have with slf4j/logback), but the basic approach is:

  • set those fields in the Mapped Diagnostic Context (I'm fairly sure log4j2 supports that)
  • use a log appender which logs to logstash-structured JSON
  • configure filebeat to ship the JSON logs
  • if filebeat is shipping to logstash, you'll need to configure logstash to pass those preformatted JSON logs directly to elasticsearch
0
votes

It is definitely possible. I am doing that now with my applications. However, the output looks a bit different from yours. The basic guide for doing this can be found at Logging in the Cloud on the Log4j2 web site.

The "normal" log view looks very similar to what you would see when logging to a file. enter image description here However, if you select a message you can see the individual fieds. enter image description here The Log4j2 configuration uses a TCP Socket appender that is configured to write to a cluster of Logstash servers that use a single DNS entry and to use the Gelf layout.

You can also use MapMessages to capture individual data elements and log them. While this currently works it is slightly cumbersome so I have recently committed improvements that will be available in Log4j 2.15.0.

It is important to note that the Logging in the Cloud page briefly mentions storing your logging configuration in Spring Cloud Config. If you want to have a common base configuration while allowing apps to do some customization this works very, very well. However, The Gelf, Json Template Layout and TCP Appender are all independent from that and can be used without Spring Boot.