How to use Stackdriver Structured Logging in App Engine Flex Java environment

Question

Google App Engine flexible environment automatically pipes stdout and stderr to Stackdriver (Google Cloud Logging). But this only supports plain text log message without any metadata (not even logging levels).

I found Logback appender for google cloud logging.

https://cloud.google.com/logging/docs/setup/java

But this does not seem to support structured logging yet. And also, it makes GRPC calls for every log entry under the hood. So, I wonder how scalable it is (especially compare to current app engine structure which has separate Fluentd agent handles logs).

Is there any out of box or simple solution to send structured log from App Engine to Stackdriver?

Have you looked at using the Stackdriver Logging Client Library? — Christopher P
@ChristopherP Yes, this is the lower level solution, and I will end up implementing Logback appender using this API. I could do it, but I assume there is a solution provided by google when there is already Logback appender they made. — essis
@essis, did you find any solution to send structured log from App Engine to Stackdriver? — Anmol Vijaywargiya

Deyvid Martinez Deyvid Martinez · Accepted Answer · 2019-10-11T21:48:37

You could use the Logback appender + Implement LoggingEnhacer

public class LogEnhancer implements LoggingEnhancer {


  @Override
  public void enhanceLogEntry(LogEntry.Builder logEntry) {
    // add Labels
    logEntry.addLabel("project", "conversational-services");

    // Transform textPayload to JSONPayload
    ObjectMapper mapper = new ObjectMapper();
    Builder structBuilder = Struct.newBuilder();
    String textPayload = logEntry.build().getPayload().getData().toString();
    try {
      // Validate JSON Payload
      mapper.readTree(textPayload);
      JsonFormat.parser().merge(textPayload, structBuilder);
      logEntry.setPayload(JsonPayload.of(structBuilder.build()));
    } catch (InvalidProtocolBufferException e) {
      // Error reporting an error! FML
      System.err.println(e.getMessage());
    } catch (IOException e) {
      // Do nothing (there is not a JSON Payload)
    }

  }

}

This class add labels and transform a JSON String in a JSONPayload

You need to specify the LoggingEnhacer on the logback.xml file

<!DOCTYPE configuration>
<configuration>
  <appender name="CLOUD" class="com.google.cloud.logging.logback.LoggingAppender">

    <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
      <level>INFO</level>
    </filter>
    <log>application.log</log> 
    <resourceType>gae_app</resourceType> 
    <!--  References to the LoggingEnhacer class -->
     <enhancer>[path_for_your_logging_enhancer_class]</enhancer>
    <flushLevel>WARN</flushLevel> 
  </appender>

  <root level="info">
    <appender-ref ref="CLOUD" />
  </root>
</configuration>

How to use Stackdriver Structured Logging in App Engine Flex Java environment

2 Answers