How does multi-line logging work in Lambda -> CloudWatch

Question

My multi-line logging events all end up multi-events - one event per line. According to the documentation:

Each call to LambdaLogger.log() results in a CloudWatch Logs event...

but then:

However, note that AWS Lambda treats each line returned by System.out and System.err as a separate event.

Looking inside LambdaAppender's source code, it seems that it proceeds to log the event to System.out anyway. So does that mean multi-line messages will always be broken down into multiple event?

I have read about configuring the multi_line_start_pattern, but that seems only applicable when you get to deploy a log agent, which isn't accessible in Lambda.

[Edit] LambdaAppender logs to LambdaLogger which logs to System.out.

[Edit] I found some post where a workaround was suggested - use '\r' for the eol when printing the messages. This seems to work for messages that my code produces. Stack traces logged everywhere are still a problem.

[Edit] I have been using two workarounds:

Log complex data structures (e.g. sizable maps) in JSON. CloudWatch actually recognizes JSON strings in log events, and pretty print them.

Replace '\n' with '\r'. For stack traces I created a utility method (this is in Kotlin, but the idea is generic enough):

fun formatThrowable(t: Throwable): String {
    val buffer = StringWriter()
    t.printStackTrace(PrintWriter(buffer))
    return buffer.toString().replace("\n", "\r")
}

I think in the long run a more ideal solution would be an Appender implementation that decorates ConsoleAppender, which would do the \r replacement on all messages passing through.

Where do you see it writing to System.out? Because I see it writing to a LambdaLogger instance, which is part of the non-distributed AWS runtime code. — kdgregory
If I read it correctly, LambdaLogger source is in the same project actually: github.com/aws/aws-lambda-java-libs/blob/master/…. — jingx
You certainly seem to be correct -- I didn't take the time to look at the other projects in the repo before commenting. However, I know (or at least believe) that I saw an exception logged in the same CloudWatch event as the main message. Which makes me wonder if maybe System.out is an AWS-specific PrintStream that flushes based on writes rather than newline. That's complete speculation on my part, but would be fairly easy to verify with (1) Class.getName() and (2) printing a string with embedded newlines. — kdgregory

aherve aherve · Accepted Answer · 2021-08-08T09:37:21

Best practice is to use json in your logs. Instead of sending multiline outputs, send a formatted json (regardless of the language you are using, you will find a lib that already does that for you)

You will be amazed how easy it gets to browse your logs from there. For instance, aws cloudwatch insights automatically detects your fields, it allow to parse them and query them within seconds

How does multi-line logging work in Lambda -> CloudWatch

2 Answers