We have an Azure Function that runs every few seconds. We want to use New Relic to let us know if this code throws exceptions or starts misbehaving. I've gotten the New Relic agent installed on the VM, and brought in the NewRelic.Agent.Api DLL through NuGet. I've set the NewRelic.AppName to AzureFunctionAppTest and set the NEWRELIC_LICENSEKEY environment variable. The log files indicate I'm connecting to NewRelic. Now, I'm running the following code:
#r "Microsoft.windowsazure.storage"
using System;
using System.Threading.Tasks;
using Microsoft.WindowsAzure.Storage.Table;
public static void Run(TimerInfo myTimer, TraceWriter log, CloudTable syncTimesTable)
{
NewRelic.Api.Agent.NewRelic.SetTransactionName("SyncEvents", "GetRoutineData");
var rnd = new Random(DateTime.Now.Millisecond).Next(30);
log.Info($"Logging {rnd}");
var eventAttributes = new Dictionary<string, object>() {{"result", "Success!"}, {"Count", rnd.ToString()}};
NewRelic.Api.Agent.NewRelic.RecordCustomEvent("DevicesSynced", eventAttributes);
NewRelic.Api.Agent.NewRelic.NoticeError(new SyncException("This is another custom error"));
}
public class SyncException : Exception
{
public SyncException(string msg) : base(msg)
{
}
}
So, basically I'm trying to set the transaction name, then I generate a random number and log it as a custom event, and then I try to log a custom error.
I let this run for a bit and then check in with the New Relic dashboard. Here's what I get:
- I see my new application show up on the APM dashboard.
- On the Insights Dashboard, I can query SELECT * FROM DevicesSynced SINCE 30 MINUTES AGO and I see a table with a bunch of records that say "Success!" with a random number in the count column.
- In APM->Events->Errors, I see a "URL and type" of "NewRelic.Api.Agent.NoticeError API Call Submission#0+SyncException" with the Message "This is another custom error" and a count.
- In Error analytics, I see "Great! You have no errors for this time window!" (Last 30 minutes)
- In Monitoring->Transactions, I only see transactions for HTTP requests such as /Home/Get, /Admin/GetHostStatus, /Keys/Get and /Admin/GetFunctionStatus. I think these HTTP requests are from inner workings of Azure Functions and maybe occur when I click around in the Azure console. I do not see any sort of transaction called GetRoutineData. If I run
SELECT * FROM TransactionsI also don't see this.
I have a feeling that I'm not actually in a transaction in my Azure Function code, so New Relic isn't letting me set a name. I also have a feeling that even though I log an error using NoticeError, it doesn't get linked to any transaction thus it doesn't show up in Error analytics.
My questions:
- Is there a way to get New Relic to recognize this code path as a transaction?
- If not, is there at least a way to set up New Relic Application Monitoring to alert me if a certain exception type (such as my SyncException class) occurs or reaches a certain thresh hold?
Thanks!
UPDATE:
I tried adding a file called CustomInstrumentation.xml in the Extensions directory, as per the New Relic instructions with the following contents:
<?xml version="1.0" encoding="utf-8"?>
<extension
xmlns="urn:newrelic-extension">
<instrumentation>
<!-- Define the method which triggers the creation of a transaction. -->
<tracerFactory name="NewRelic.Agent.Core.Tracer.Factories.BackgroundThreadTracerFactory" metricName="Background/AzureDeviceSync">
<match assemblyName="Microsoft.Azure.WebJobs.Script" className="Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase">
<exactMethodMatcher methodName="Invoke" />
</match>
</tracerFactory>
</instrumentation>
</extension>
However, I don't see any difference in behavior. I've tried a few different method name matches (such as InvokeCore and Invoke) however it doesn't appear to make a difference.
Few questions:
The New Relic Docs says:
To view the transaction: From the New Relic menu bar, select APM > Applications > (selected app) > Monitoring > Transactions > Type > (selected type). (The type is defined by Category/Name.) You can then view all custom transactions of the selected type.
It shows a picture of a dropdown menu with a Custom option:
However, my New Relic interface doesn't have any sort of "Type" dropdown:
Also, if I instrument my code to show a stack trace from within the Function app, here's what I get. Maybe it'll help:
Stack:
at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
at System.Environment.get_StackTrace()
at Submission#0.Run(TimerInfo myTimer, TraceWriter log, CloudTable syncTimesTable) in :line 9
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at Microsoft.Azure.WebJobs.Script.Description.DotNetFunctionInvoker.<InvokeCore>d__23.MoveNext()
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[TStateMachine](TStateMachine& stateMachine)
at Microsoft.Azure.WebJobs.Script.Description.DotNetFunctionInvoker.InvokeCore(Object[] parameters, FunctionInvocationContext context)
at Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.<Invoke>d__29.MoveNext()
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[TStateMachine](TStateMachine& stateMachine)
at Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters)
at Host.Functions.LogNewRelicAlert(TimerInfo myTimer, TraceWriter log, CloudTable syncTimesTable, ExecutionContext _context)
at lambda_method(Closure , Functions , Object[] )
at Microsoft.Azure.WebJobs.Host.Executors.TaskMethodInvoker`1.InvokeAsync(TReflected instance, Object[] arguments)
at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`1.<InvokeAsync>d__8.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.InvokeMoveNext(Object stateMachine)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run()
at System.Runtime.CompilerServices.YieldAwaitable.YieldAwaiter.RunAction(Object state)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
I'm wondering if there's any sort of log file that shows if custom transaction settings are indeed loaded and when a new transaction is created. Right now I'm guessing my XML is matching the wrong thing.
One more thing. If I add the line:
NewRelic.Api.Agent.NewRelic.SetTransactionName("Custom", "Testing");
In my Function app, I'll see the following warning in the New Relic logs:
2017-02-01 21:45:25,005 NewRelic WARN: Agent API Error: An error occurred invoking API method "SetTransactionName" - "System.InvalidOperationException: The API method called is only valid from within a transaction. This error can occur if you call the API method from a thread other than the one the transaction started on.
I think that makes it clear we're not actually in a transaction at this point, unless it's happening on another thread (which I wouldn't be surprised if this were the case).
Other ideas?
