Istio + Jaeger tracing with gRPC calls using Spring Boot

Question

I'm using this library (grpc-spring-boot-starter) so I can have gRPC capabilities in a Spring Boot app.

I want to know how to properly integrate this with Istio + Jaeger tracing.

I need to know what are the needed dependencies for this to happen.

I have two (2) apps, one serves as the gRPC client and one serves as the gRPC server,

The expectation is that the trace between the gRPC client and the gRPC server must be reflected in Jaeger. But it's not happening.

I am inside a Kubernetes cluster that has Istio.

What really happens is HTTP request -> Envoy sidecar -> gRPC Client's Spring Boot @RestController -> get the Headers from HTTP request -> copy those to gRPC call before making the call -> gRPC Service.

How can I make the gRPC client <-> gRPC Service trace shown to Jaeger?

Are there any dependencies that needs to be imported?

Right now I have:

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-sleuth</artifactId>
</dependency>
<!-- Used together with Sleuth to be able to trace gRPC calls. -->
<dependency>
    <groupId>io.zipkin.brave</groupId>
    <artifactId>brave-instrumentation-grpc</artifactId>
</dependency>
<!-- Add'l dependency for Istio + Jaeger gRPC tracing -->
<dependency>
    <groupId>io.jaegertracing</groupId>
    <artifactId>jaeger-client</artifactId>
</dependency>

I also have done something like this to "propagate the headers":

private void propagateHeaders(HttpHeaders headers) {
    // TODO: adding these string values to a final / constant.
    // get these tracing headers from HTTP request, (coming from Envoy)
    String xRequestIdFromHttpHeader = headers.getFirst("x-request-id");
    String xB3TraceIdFromHttpHeader = headers.getFirst("x-b3-traceid");
    String xB3SpanIdFromHttpHeader = headers.getFirst("x-b3-spanid");
    String xB3ParentSpanIdFromHttpHeader = headers.getFirst("x-b3-parentspanid");
    String xB3SampledFromHttpHeader = headers.getFirst("x-b3-sampled");
    String xB3FlagsFromHttpHeader = headers.getFirst("x-b3-flags");
    String xOtSpanContextFromHttpHeader = headers.getFirst("x-ot-span-context");

    // create a custom gRPC header, they call it Metadata
    Metadata xRequestIdMetadata = new Metadata();
    Metadata xB3TraceIdMetadata = new Metadata();
    Metadata xB3SpanIdMetadata = new Metadata();
    Metadata xB3ParentSpanIdMetadata = new Metadata();
    Metadata xB3SampledMetadata = new Metadata();

    // TODO: refactor. putting to List<> and using foreach to attach this one.
    // assign value of the x-request-id Metadata to the value of the HTTP Header
    xRequestIdMetadata.put(Metadata.Key.of("x-request-id", Metadata.ASCII_STRING_MARSHALLER), xRequestIdFromHttpHeader);
    xB3TraceIdMetadata.put(Metadata.Key.of("x-b3-traceid", Metadata.ASCII_STRING_MARSHALLER), xB3TraceIdFromHttpHeader);
    xB3SpanIdMetadata.put(Metadata.Key.of("x-b3-spanid", Metadata.ASCII_STRING_MARSHALLER), xB3SpanIdFromHttpHeader);
    xB3ParentSpanIdMetadata.put(Metadata.Key.of("x-b3-parentspanid", Metadata.ASCII_STRING_MARSHALLER), xB3ParentSpanIdFromHttpHeader);
    xB3SampledMetadata.put(Metadata.Key.of("x-b3-sampled", Metadata.ASCII_STRING_MARSHALLER), xB3SampledFromHttpHeader);

    // TODO: refactor. putting to List<> and using foreach to attach this one.
    // use MetadataUtils to attach that new Metadata to our stub before requesting to our gRPC Server via gRPC
    greetingServiceStub = MetadataUtils.attachHeaders(greetingServiceStub, xRequestIdMetadata);
    greetingServiceStub = MetadataUtils.attachHeaders(greetingServiceStub, xB3TraceIdMetadata);
    greetingServiceStub = MetadataUtils.attachHeaders(greetingServiceStub, xB3SpanIdMetadata);
    greetingServiceStub = MetadataUtils.attachHeaders(greetingServiceStub, xB3ParentSpanIdMetadata);
    greetingServiceStub = MetadataUtils.attachHeaders(greetingServiceStub, xB3SampledMetadata);
}

But it doesn't seem to work..

Answer 1

What you did is for http 1.x, and it doesn't work for http2/grpc. Please dive into grpc impl in springboot doc.