Azure Worker role error handling

Question

working with worker roles is a little bit tricky if something goes wrong. I have exception occurring in my worker role, which forces role to quit and restart.

I decided to implement azure diagnostics solution to log what is going on. It works, when i try to write message to trace, but i am unable to catch exceptions and log them. I surround my code with try catch and try to write exception contents to trace.

Is this right approach? And why does it not work? Is there any better way to log exceptions in worker roles? Thank you. My code is:

public class WorkerRole : RoleEntryPoint
    {

    private const int WAIT_INTERVAL_SECONDS = 15;

    public override void Run()
    {
        Trace.WriteLine("$projectname$ entry point called", "Information");

        while (true)
        {

            try
            {
                string id = MainWCFRole.Storage.TrackProcessingQueueDAO.retrieveMsgContents();
                if ((id != null) && (!id.Equals("")))
                {
                    var points = MainWCFRole.Storage.TrackRawDataDAO.retrieve(id);

                    Processor.process(id, points);
                }
                else
                {
                    Thread.Sleep(WAIT_INTERVAL_SECONDS * 1000);
                }
            }
            catch (Exception ex)
            {
                Trace.TraceInformation("Something went wrong");
                Trace.TraceError(ex.ToString());
                throw ex;
            }


        }
    }

    public override bool OnStart()
    {
        // Set the maximum number of concurrent connections 
        ServicePointManager.DefaultConnectionLimit = 12;

        DiagnosticMonitorConfiguration dmc = DiagnosticMonitor.GetDefaultInitialConfiguration();
        dmc.Logs.ScheduledTransferPeriod = TimeSpan.FromMinutes(1);
        dmc.Logs.ScheduledTransferLogLevelFilter = LogLevel.Verbose;
        dmc.WindowsEventLog.ScheduledTransferLogLevelFilter = LogLevel.Verbose;
        dmc.WindowsEventLog.ScheduledTransferPeriod = TimeSpan.FromMinutes(1);


        DiagnosticMonitor.Start("StorageConnectionString", dmc);

        Trace.TraceInformation("Starting Worker Role TrackProcessor");

        return base.OnStart();
    }

}

Processor.process(id, points) is method which throws exception. More precisely, it is when i try to call SQL azure query. I know that diagnostics works, because when role starts, it calls Trace.TraceInformation("Starting Worker Role TrackProcessor"); and it is visible in the table storage as mesage. Then over time more messages occur, as role encounters exception and is forced to restart. However, no error messages in log.

Can you try adding a delay before you throw your exception? It might be that your role is getting recycled before the monitor agent on the node has a chance to upload the logs to diagnostics. — David Z.
Sounds promising, i am going to try. Is there way to force Diagnostics monitor to persist data on demand instead of on schedule? Could be resolved instantly and role restarted immediatly. — user1018711

Brian Reischl Brian Reischl · Accepted Answer · 2012-04-24T18:20:16

If the Run() method exits, then your worker is recycled. The worker may not have time to transfer logs to storage before it gets recycled.

The simplest solution is to not re-throw the exception after you catch it. Then your worker role will continue running until it is shut down manually.

Azure Worker role error handling

2 Answers