Issue sending traces from a docker container to a Jaeger docker container running in a VM

Question

I've added opentracing to my web app and am using the Jaeger all-in-one docker image as the collector. I'm running docker on windows 10 (hyper-v) and am using the Jaeger java client. When I test the web app locally on the host machine it sends traces successfully to the Jaeger collector docker instance. However, when I run the web app in another docker container no traces are registered in the Jaeger UI.

I've tried both containers on the same docker network with no success. The web app in the docker container can access other services in dockers contains such as a DB and an ETCD server with no issues.

I thought I might have the wrong ports but given the fact it works from the host environment I'm assuming these are correct and that it is a docker configuration issue. I've also set JAEGER_SAMPLER_TYPE environment variable to const and the JAEGER_SAMPLER_PARAM to 1 to ensure all traces are logged.

Edit - when I look at the metrics it seems like the jaeger is receiving the spans. Every call I make to the app increments this count by one.

jaeger_spans_received_total{debug="false",format="proto",svc="datastore",transport="grpc"} 35
jaeger_spans_saved_by_svc_total{debug="false",result="ok",svc="datastore"} 35
jaeger_traces_received_total{debug="false",format="proto",sampler_type="const",svc="datastore",transport="grpc"} 35
jaeger_traces_saved_by_svc_total{debug="false",result="ok",sampler_type="const",svc="datastore"} 35

I also tried the sample project Hotrod as suggested by Yuri Shkuro on a similar issue someone had. Exact same result as above. Metrics shows spans being received but nothing is displayed in UI. Edit 2 - I've narrowed it down to happening in a hyper-v windows 10 VM.

Any help would be appreciated.

Thanks

192.168.0.15 is host machine

# deployment of jaeger
 docker run --name jaeger-collector -d -p 5775:5775/udp -p 6831:6831/udp -p 6832:6832/udp -p 5778:5778 -p 16686:16686 -p 14268:14268 --network mynetwork jaegertracing/all-in-one:latest

# deployment of app 
 docker run --add-host etcd-01:192.168.0.15 --name datastore -d -it --rm -p 8030:8080 --network mynetwork datastore:latest

Setting up Jaeger tracer in Java

 private static io.opentracing.Tracer getJaegerTracer() {

        Configuration.SamplerConfiguration samplerConfig = Configuration.SamplerConfiguration.fromEnv().withType("const").withParam(1);

        Configuration.SenderConfiguration senderConfig = Configuration.SenderConfiguration.fromEnv()
                    .withAgentHost("192.168.0.15")
                    .withAgentPort(6831);

        Configuration.ReporterConfiguration reporterConfig = Configuration.ReporterConfiguration.fromEnv().withLogSpans(true).withSender(senderConfig);
        Configuration config = new Configuration(DatastoreConstants.SERVICE_NAME).withSampler(samplerConfig).withReporter(reporterConfig);


        return config.getTracer();
    }

I wonder if you have clock issue where the timestamps reported in the spans are different from what UI thinks the current time is, and therefore the time window that UI sends to the server does not match the span data. — Yuri Shkuro

Schaffer Schaffer · Accepted Answer · 2020-06-10T09:52:33

Thanks Yuri. Yes it was a clock issue. Although the host machine (VM) updated its clock on every unpause, docker for windows did not. The timezones were correct for all containers but the times were all out by the exact same amount. This must be the docker internal clock that seems to only get updated once on launch and not at the launch of every new container. Although all container clocks were out by the same amount, the windows host machine was correct. The messages were arriving but the times/dates were outside the time frame window the UI was displaying. If I set a custom date range I'm sure they would appear. The containers must have been out by a few days with the continual stopping and starting and not by just a few hours.

Issue sending traces from a docker container to a Jaeger docker container running in a VM

2 Answers