Distributed tracing with Elastic APM on .NET Core, non-HTTP

Question

I am interested in using Elastic APM within an ASP.NET Core to instrument traces of a set of services which communicate over a mix of protocols (HTTP, SQS, SNS). Despite reviewing the documentation, I am not clear how I can use the Elastic APM Public API to connect transactions to one another which occur outside of HTTP (HttpClient is automatically instrumented for trace by Elastic APM).

According to the documentation, I should be able to serialize the CurrentTransaction.OutgoingDistributedTracingData on the caller and then deserialize it to resume the transaction on the callee, but despite implementing this pattern in memory, my traces in Kibana are missing spans from all but the final transaction.

// transaction 1
var trans1 = Agent.Tracer.StartTransaction("Dist Trans 2", ApiConstants.TypeRequest);

await trans1.CaptureSpan("step 1 processing", ApiConstants.ActionExec, async () => await Task.Delay(30));

// transaction 2
var trans2 = Agent.Tracer.StartTransaction("Dist Trans 2", ApiConstants.TypeRequest,
    DistributedTracingData.TryDeserializeFromString(trans1.OutgoingDistributedTracingData.SerializeToString()));

await trans2.CaptureSpan("step 2 processing", ApiConstants.ActionExec, async () => await Task.Delay(30));

// transaction 3
var trans3 = Agent.Tracer.StartTransaction("Dist Trans 2", ApiConstants.TypeRequest,
    DistributedTracingData.TryDeserializeFromString(trans2.OutgoingDistributedTracingData.SerializeToString()));

await trans3.CaptureSpan("step 3 processing", ApiConstants.ActionExec, async () => await Task.Delay(30));

trans3.End();

My implementation spike can be found on Github.

gregkalapos gregkalapos · Accepted Answer · 2020-03-27T18:53:59

You don't end trans1 and trans2.

Just put these 2 lines to the point where these end, and everything should show up fine:

trans1.End();
trans2.End();

There is the CaptureTransaction, which is a convenient method that can wrap you any code and makes sure the transaction is ended and all exceptions are captured - so you use that method and it does "everything" for you.

Then there is the StartTransaction method - this is the one you use in your code -, which starts the transaction and does not do anything else. The advantage here is that you get an ITransaction instance which you can use wherever and whenever you want. But in this case you need to call .End() on it manually once the transaction (aka the code you want to capture) is executed.

Same with CaptureSpan and StartSpan.

So you used CaptureSpan for your spans, so those where ended automatically when the lambda with Task.Delay finished, on the other hand you started your transactions with StartTransaction but only called .End() on trans3 and not on the 2 other transactions.

There is some explanation with a demo here - sample code of that demo is here.

Distributed tracing with Elastic APM on .NET Core, non-HTTP

2 Answers