AWS streaming multi-line log files from CloudWatch to ELK

Question

We are streaming app logs from CloudWatch to AWS ELK. Our microservices are written in Java and so I am only concentrating on those. A typical java exception stack trace when logged looks like this:

Exception in thread "main" java.lang.NullPointerException
    at com.example.myproject.Book.getTitle(Book.java:16)
    at com.example.myproject.Author.getBookTitles(Author.java:25)
    at com.example.myproject.Bootstrap.main(Bootstrap.java:14)

Normally, this will be ingested line by line in ELK stack, which breaks the entire message.

Usually, For entire stack trace to be ingested as a single message, one can configure multiline plugin either in Logstash or Filebeat.

Any idea how to enable multiline while streaming log files from CloudWatch to ELK by AWS lambda?

What is your logging framework, and do you have any options to reconfigure it? — kdgregory
And how are you writing the logs to CloudWatch? Are you running in Lambda and writing to its logger, or are you running in a container (or on EC2) and using the CloudWatch log agent? If the latter, then your problem is probably that agent, because it breaks the message into multiple lines in the log. — kdgregory
@kdgregory slf4j with default config are used. any suggestions how to configure it? we run app in a container and use the CloudWatch log agent. is it possible to tell CloudWatch agent not break log per-line and combine them in case of sracktrace — Fedor

Fedor Fedor · Accepted Answer · 2018-11-10T21:40:58

It turned out to be a know issue with AWS cloudwatch logger, see RichardBronosky's comment at https://github.com/visionmedia/debug/issues/296

I did some testing and found that CloudWatch log entries can be made multiline by using \r as the line delimiter. Using either \n (Unix) or \r\n (DOS) line endings will result in separate entries

So I fixed it by adding following ExceptionHandler to Spring Boot RestController

@ExceptionHandler(Throwable.class)
void handleUnhandledExceptions(Throwable e, HttpServletResponse response) throws IOException {
    StringWriter buffer = new StringWriter();
    e.printStackTrace(new PrintWriter(buffer));
    log.error(buffer.toString().replace("\n", "\r"));
    response.sendError(HttpStatus.INTERNAL_SERVER_ERROR.value(), e.getMessage());
}

AWS streaming multi-line log files from CloudWatch to ELK

1 Answers