Spring cloud sleuth adding tag

Question

I'm trying to implement distributed tracing in my kotlin app using spring cloud sleuth. I'm sending those data to the datadog. Now I'm able to trace my logs but I want to add some extra data to spans. Let's say I want to add info about user and be able to see it in datadog. Am I right that span tags are good for it? I'm sending the logs in json format to datadog but I cannot add tags here. (traceId and spanId are injected). Logback config:

        <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
            <providers>
                <timestamp/>
                <version/>
                <message/>
                <loggerName/>
                <threadName/>
                <logLevel/>
                <logLevelValue/>
                <callerData/>
                <stackTrace/>
                <rootStackTraceElement/>
                <context/>
                <mdc/>
                <tags/>
                <logstashMarkers/>
                <arguments/>
            </providers>
        </encoder>

gradle:

implementation("org.jetbrains.kotlin:kotlin-reflect")
implementation("org.jetbrains.kotlin:kotlin-stdlib-jdk8")
implementation("org.springframework.cloud:spring-cloud-starter-netflix-eureka-server")
implementation("org.springframework.cloud:spring-cloud-starter-vault-config")
implementation("org.springframework.cloud:spring-cloud-starter-sleuth")
implementation("org.springframework.boot:spring-boot-starter-actuator")
implementation("ch.qos.logback:logback-classic:1.2.3")
implementation("net.logstash.logback:logstash-logback-encoder:6.6")
implementation("org.zalando:logbook-spring-boot-starter:2.4.2")
developmentOnly("org.springframework.boot:spring-boot-devtools")
runtimeOnly("io.micrometer:micrometer-registry-datadog")
testImplementation("org.springframework.boot:spring-boot-starter-test")

and to add the tag I'm trying

@NewSpan
fun newSpanTest() {
    tracer.currentSpan()!!.tag("user", "123")
    log.info("other span")
    otherTestService.sameSpanTest()
}

example log:

{"@timestamp":"2021-03-09T21:04:46.953+01:00","@version":"1","message":"other span","logger_name":"com.microservices.text.rpg.servicediscovery.TestService","thread_name":"http-nio-8761-exec-3","level":"INFO","level_value":20000,"caller_class_name":"com.microservices.text.rpg.servicediscovery.TestService","caller_method_name":"newSpanTest","caller_file_name":"TestService.kt","caller_line_number":25,"traceId":"e61fd165d7c84776","spanId":"5e61f9b51b51619b"}

shouldn't be that 'user' injected into MDC and then into logs?

Jonatan Ivanov Jonatan Ivanov · Accepted Answer · 2021-03-09T21:11:35

TL;DR

spring.sleuth.baggage.correlation-fields automatically sets baggage values to Slf4j’s MDC so you only need to set the baggage field.

Longer version with background

I suppose you use Sleuth out of the box (uses Brave):

Fist get familiar with tag, baggage and their differences; you can also read about them in Brave docs
Check Brave's ScopeDecorator, CorrelationScopeDecorator and MDCScopeDecorator

The spring.sleuth.baggage.correlation-fields property automatically sets baggage values to Slf4j’s MDC so you only need to set the baggage field.

Also, using MDCScopeDecorator, you can set the baggage values to Slf4j’s MDC programmatically, you can see how to do it in Sleuth docs:

// configuration
@Bean
BaggageField countryCodeField() {
    return BaggageField.create("country-code");
}

@Bean
ScopeDecorator mdcScopeDecorator() {
    return MDCScopeDecorator.newBuilder()
            .clear()
            .add(SingleCorrelationField.newBuilder(countryCodeField())
                    .flushOnUpdate()
                    .build())
            .build();
}

// service
@Autowired
BaggageField countryCodeField;

countryCodeField.updateValue("new-value");

Spring cloud sleuth adding tag

1 Answers

TL;DR

Longer version with background