Integrating AWS Java SDK 2.x with Spring Sleuth

Question

I'm working with using Spring Boot for the first time and am setting up distributed tracing for my application. I've added Spring Cloud Slueth to my app and I could see the spans and traces being generated when my endpoints are being called, but I'm trouble getting it to integrate with Aws Sdk 2.x (using the Dynamo async client). I have a few questions about integrations:

1. What is the best way to a trace http calls through the aws sdk. The only way I've been able to do it is to implement the ExecutionInterceptor and manually create a span:

@Component
@Log4j2
public class AwsExecutionInterceptor implements ExecutionInterceptor {
    public static final String CLIENT_CALL_SPAN = "ClientCallSpan";
    public static final ExecutionAttribute<Span> ATTRIBUTE = new ExecutionAttribute<>(
            CLIENT_CALL_SPAN);
    private final Tracer tracer;

    @Autowired
    public AwsExecutionInterceptor(Tracer tracer) {
        this.tracer = tracer;
    }

    @Override
    public void beforeExecution(Context.BeforeExecution context, ExecutionAttributes executionAttributes) {

        Span clientCallSpan = this.tracer.nextSpan().name("clientCall").start();
        log.info("Starting new span");
        executionAttributes.putAttribute(ATTRIBUTE, clientCallSpan);
    }

    @Override
    public void afterExecution(Context.AfterExecution context, ExecutionAttributes executionAttributes) {
        Span span = executionAttributes.getAttribute(ATTRIBUTE);
        span.end();
        log.info("Span closed");
    }
}

2. Is there a way to get trace ids to propagate to the the logs from the netty threadpool that are logged via the aws sdk? I am providing an executor via the advanced options, but traces aren't propagated to logs from netty threads. Here is an example:

public class AwsExecutorConfig {

    @Bean("AwsClientExecutor")
    public Executor executor() {

        ThreadPoolExecutor executor = new ThreadPoolExecutor(50, 50,
                                                             10, TimeUnit.SECONDS,
                                                             new LinkedBlockingQueue<>(10_000));

        executor.allowCoreThreadTimeOut(true);
        return executor;
    }
}

@Autowired
public DynamoConfig(AwsExecutionInterceptor executionInterceptor,
                    @Qualifier("AwsClientExecutor") Executor executor) {

  @Bean(destroyMethod = "close")
  public DynamoDbAsyncClient dynamoDbAsyncClient() {
    var builder = DynamoDbAsyncClient.builder();
    builder.overrideConfiguration(
        ClientOverrideConfiguration.builder()
                                   .addExecutionInterceptor(executionInterceptor).build());

    builder.asyncConfiguration(
            b -> b.advancedOption(SdkAdvancedAsyncClientOption.FUTURE_COMPLETION_EXECUTOR, 
                                  executor));
    return builder.build();
  }
}

With this, I can see logs like the following from the sdk,

2021-07-20 10:26:04.369 DEBUG [,c9ae653fd2254266,c9ae653fd2254266] 2873 --- [pool-2-thread-5] s.a.a.m.internal.DefaultMetricCollector  : Collected metrics records: MetricCollection(...)

but other logs don't have any traces, for example:

2021-07-20 10:26:04.360 DEBUG [,,] 2873 --- [tyEventLoop-0-2] software.amazon.awssdk.request           : Received successful response: 200
2021-07-20 10:26:04.361 TRACE [,,] 2873 --- [tyEventLoop-0-2] software.amazon.awssdk.request           : Parsing service response JSON.
2021-07-20 10:26:04.361 DEBUG [,,] 2873 --- [tyEventLoop-0-2] software.amazon.awssdk.requestId         : x-amzn-RequestId : af0d9968-3202-480f-9e3c-8312ae6963c8
2021-07-20 10:26:04.361 DEBUG [,,] 2873 --- [tyEventLoop-0-2] software.amazon.awssdk.requestId         : x-amz-id-2 : not available
2021-07-20 10:26:04.367 TRACE [,,] 2873 --- [tyEventLoop-0-2] software.amazon.awssdk.request           : Done parsing service response.

The spring cloud sleuth docs mention integration with the Netty Http client, but I'm not sure how to do this with the AWS SDK. Any help with this would be greatly appreciated

Answer 1

This is a pretty loaded question, let me try to give you some background and explain what you can do/look for.

Spring Cloud Sleuth

Spring Cloud Sleuth is an abstraction over distributed tracing libraries (like OpenZipkin's Brave). This means that you can use Sleuth's API to instrument your code and you can change your tracing library under the hood. This also means that you can use the built-in instrumentations of the tracing library of your choice (e.g.: you instrument your code using Sleuth but you can also add Brave's MySQL instrumentation and the two things will work together).

(Brave is not the only tracing library supported by Sleuth. Sleuth also has OTel support but that one is not GA yet so we are not recommending it in production.)

How to find instrumentation support

Because of the things above, you can look instrumentation support in a couple of places:

1. Spring Cloud Sleuth instruments a bunch of things that are supported by Spring Framework (AWS is not like that, Spring Cloud AWS is a community-supported project now) but as you said the netty client is: docs
2. Brave has instrumentation support for a lot of libraries, you can take a look there if there is anything you can reuse (e.g.: there is some AWS support in a different repo)
3. The libraries themselves can have Brave support (I don't think this is the case for the AWS SDK as of today)

How to instrument things yourself

I recommend using Sleuth API to instrument your code, here you can find how: Creating and Ending Spans. As you can see, this can be tricky if you want to instrument a third-party library, there a couple of things that you can do though:

1. Something similar to what you did above with the interceptor (this assumes that there is such a component that you can use to inject your own code). Fyi: your implementation does not create the "scope", [see the docs](Creating and Ending Spans).
2. You can utilize AOP, create an Aspect and inject the instrumentation logic that way (search for "aspect" here).
3. You can create a wrapper that will do the instrumentation and delegate to the instrumented component. You can wrap the original component by creating a bean or by using a BeanPostProcessor (search for "beanpostprocessor" here).

Context Propagation to logs

The log integration usually happens through putting the tracing data into the MDC and tracing data is usually propagated through thread-locals. Because of this, if you change threads during the execution, your tracing data is lost and being inside of the thread-pool is like starting a completely new span from scratch (no parent span). Fortunately Sleuth has a solution for this, you need to wrap your Executor into a component that Sleuth provides and Sleuth will propagate the context into your thread-pool for you, see: thread-pool support

Issues with Netty's HttpClient

Please check the HttpClient part of the docs, it needs to be a bean so that Sleuth can instrument it.