Spring cloud Sleuth starts a new trace instead of continuing spans in a single trace

Question

I have 4 spring-boot applications (A, B, C and D). The lifecycle of a transaction is as follows :

Application A is a kafka streams application and it ultimately produces to a topic which is consumed by Application B.
Application B then consumes from the topic using @KafkaListener, does some processing and then produces to IBMMQ queue using spring's jmsTemplate.
Application C which is a @JMSListener consumes from the above queue and produces to another queue using spring's JMSTemplate.
Application D which is again a @JmsListener consumes from the above queue and then produces to a kafka topic, which the again consumed by Application A

Now for a single transaction I would expect a single trace across all four application, but instead I get

One Trace starting from application A to application B (where it produces to IBM MQ)
One trace starting from Application C and ending at Application A

I would have uploaded the pictures to show the zipkin spans, but for some reason I am not able to do so.

All the above applications are Spring boot applications and they utilize spring-cloud-sleuth for producing transactions traces. I am relying on spring boot's autoconfiguration and these are the properties that I have set in all the applications:

  zipkin:
   enabled: ${ZIPKIN_ENABLED:false}
   sender:
    type: kafka
   baseUrl: ${ZIPKIN_URL:http://localhost:9411}
   service:
    name: ${spring.application.name}
 sleuth:
  messaging:
   kafka:
    enabled: true
   jms:
    enabled: true

I am not able to understand what's exactly happening here. Why the spans are scattered across 2 traces and not one?

I am using spring-boot 2.3.3 and spring-cloud-dependencies Hoxton.SR8.

Most likely that's a bug then related to the JMS template integration. We're using OpenZipkin Brave for that so either it's a bug in Sleuth or Brave. You'd have to debug this to see if the tracing headers are set in the outbound message or are they not propagated on the recipient side. — Marcin Grzejszczak
I can see that when Application B produces a message to IBMMQ, it does so with b3 property, something like b3=00d37562420424e5-b78b8a0a2b6eb173-1, but when Application C consumes it, it's missing from the message. On the other hand when ApplicationC produces to IBMMQ, it has a similar property in the message, and when Application D consumes it, the property is still there. The configuration is exactly same for all the spring JMS components across all the applications. Seems like something very subtle, that I have missed somewhere. — user3244615
You could try to debug the application c and its recipient instrumentation code to see why the header is not parsed — Marcin Grzejszczak

user3244615 user3244615 · Accepted Answer · 2020-11-15T13:54:52

So it was application B which was not passing the header along. Turns out that the queue uri had a property targetClient which was set to 1. The uri is something like

queue:///DESTINATION_QUEUE?targetClient=1

Now I am not an IBM MQ expert by far, but the documentation states that setting this property to 1 means that Messages do not contain an MQRFH2 header. I toggled it to 0 and voila, all spans fall into place.

Spring cloud Sleuth starts a new trace instead of continuing spans in a single trace

1 Answers