Could not trace spring-cloud-stream Listener with spring-cloud-sleuth

Question

I have 2 very simple spring-cloud-stream applications. Service3, the message producer, sends messages to Service4, the consumer, through the binder-kafka.

And I use spring-cloud-sleuth to trace the spans among them. But only the spans in Service3 are available in zipkin server. No span shows for Service4.

1. Service3

   dependencies {
   compile('org.springframework.boot:spring-boot-starter-web')
   compile('org.springframework.boot:spring-boot-starter-thymeleaf')
   
   compile('org.springframework.cloud:spring-cloud-starter-sleuth')
   
   // Marshal spans over a Spring cloud stream binder
   compile('org.springframework.cloud:spring-cloud-sleuth-stream')
   compile('org.springframework.cloud:spring-cloud-stream-binder-kafka')
   
   testCompile group: 'junit', name: 'junit', version: '4.11'
   } 
   
   
   @SpringBootApplication
   public class Service3  {
       public static void main(String[] args) {
           SpringApplication.run(Service3.class, args);
       }
   }
   
   
   @Controller
   @EnableBinding(Source.class)
   public class WebController {
   
       @Autowired
       private Source source;
   
   
       @GetMapping("/srv4")
       private String getSrv4Info(){
           String msg = "Hello Service 4";
           this.source.output().send(MessageBuilder.withPayload(msg).build());
   
           return "srv4";
       }
   }
   

2. Service4

   dependencies {
       compile('org.springframework.boot:spring-boot-starter-web')
   
       compile('org.springframework.cloud:spring-cloud-starter-sleuth')
   
       // Marshal spans over a Spring cloud stream binder
       compile('org.springframework.cloud:spring-cloud-sleuth-stream')
       compile('org.springframework.cloud:spring-cloud-sleuth-zipkin-stream')
       compile('org.springframework.cloud:spring-cloud-stream-binder-kafka')
   
       runtime('io.zipkin.java:zipkin-autoconfigure-ui')
   
       testCompile group: 'junit', name: 'junit', version: '4.11'
   }
   
   @SpringBootApplication
   @EnableZipkinStreamServer
   public class Service4 {
       public static void main(String[] args) {
           SpringApplication.run(Service4.class, args);
       }
   }
   
   @EnableBinding(Sink.class)
   public class MsgReceiver {
       @StreamListener(Sink.INPUT)
       private void listen(Message<String> msg){
           System.out.println(msg.getPayload());
       }
   }
   

Servic4 (message consumer) is not traced

What did I miss?

Answer 1

This is a guess.

Kafka has no concept of message headers (where spans are stored).

SCSt therefore has to embed message headers in the payload.

The current version requires you to "opt-in" which headers you want transported in this way.

Documentation here.

spring.cloud.stream.kafka.binder.headers

The list of custom headers that will be transported by the binder.

Default: empty.

Unfortunately, patterns are not currently supported, you have to list the headers individually. We are considering adding support for patterns and/or transporting all headers by default.

Answer 2

Finally, I found 2 issues related with my applications. 1. The application with @EnalbeZipkinStreamServer could not be traced. This looks like a by design. 2. If kafka is used as the binder, the applications should specify the headers as the following:

    spring.cloud.stream.kafka.binder.headers[0]=spanId
    spring.cloud.stream.kafka.binder.headers[1]=spanSampled
    spring.cloud.stream.kafka.binder.headers[2]=spanProcessId
    spring.cloud.stream.kafka.binder.headers[3]=spanParentSpanId
    spring.cloud.stream.kafka.binder.headers[4]=spanTraceId
    spring.cloud.stream.kafka.binder.headers[5]=spanName
    spring.cloud.stream.kafka.binder.headers[6]=spanFlags