Sleuth instrumentation of Spring cloud stream messages is lost when using zipkin + kafka

Question

I have the following setup :

several Spring Cloud Edgware services with most notably the spring-cloud-starter-zipkin and spring-cloud-stream-binder-kafka dependencies.
a Zipkin server running the openzipkin/zipkin:2.8 docker image
I see traces from the service going into the "zipkin" topic, zipkin picks those traces and I can look at them in the zipkin ui.
everything seems OK at first sight, especially for rest call between my services, but I just saw that it doesn't work for my own messages that I use for asynchronous communications between my services (I use spring cloud stream for this).
I can see the log messages from TraceChannelInterceptor indicating that the instrumentation is taking place
but the trace message headers are not present in the message when I inspect the data in kafka.

There used to be a StreamEnvironmentPostProcessor that did the job of adding the trace headers to the kafka bindings when I included the spring-cloud-sleuth-stream dependendy in the past. But the doc clearly states now :

Note: spring-cloud-sleuth-stream is deprecated and incompatible with these destinations

What should I do to make this work properly now ? Add the headers to the bindings configuration myself ? Or is there something I'm missing ?

I think we need more information. Spring Cloud Sleuth Stream was used to send messages to Zipkin not to instrument messages. Message instrumentation should still work. According to the Stream documentation there should be no need to set the header configuration anymore, but you can try to set the headers explicitly like here github.com/spring-cloud-samples/brewery/blob/master/brewing/src/… and tell us if that helped. If not, then we'll need a sample I guess. — Marcin Grzejszczak
Yes message instrumentation is still working (I see the log messages from TraceChannelInterceptor), but the trace headers that TraceChannelInterceptor is adding are lost immediately because these headers are not properly declared in the bindings anymore (this job was done by StreamEnvironmentPostProcessor previously). If I add the configuration myself (exactly like in the links you pointed to) it works all right. — Sébastien Nussbaumer
I can do that manually, not a problem, it's just a bit surprising that it's something that used to work out of the box, and suddenly you have to declare 7 headers manually. — Sébastien Nussbaumer
But maybe it was just a nice side effect, not intended at all, when using spring-cloud-sleuth-stream in the past — Sébastien Nussbaumer
That shouldn't be the case (the necessity to add headers) at all in the first place. Can you file an issue in github.com/spring-cloud/spring-cloud-sleuth/issues ? — Marcin Grzejszczak

Marcin Grzejszczak Marcin Grzejszczak · Accepted Answer · 2018-06-11T13:23:06

That was a bug in Spring Cloud Sleuth in Edgware. The Stream Kafka Binder in Edgware required explicit passing of headers that should get propagated. The side effect of adding sleuth-stream on the classpath was exactly that feature. By fixing the https://github.com/spring-cloud/spring-cloud-sleuth/issues/1005 issue we're adding back the missing feature to core. This is not ported to Finchley since Stream Kafka Binder in Finchley passes all headers by default.

The workaround for Edgware is to pass a list of headers in the following manner:

spring:
  cloud:
    stream:
      kafka:
        binder:
          headers:
            - spanId
            - spanSampled
            - spanProcessId
            - spanParentSpanId
            - spanTraceId
            - spanName
            - messageSent

Sleuth instrumentation of Spring cloud stream messages is lost when using zipkin + kafka

1 Answers