How to check running status and stop Durable function

7
votes

I want to process millions of records on-demand, which takes approximate 2-3 hours to process. I want to go serverless that is why tried durable function (first-time). I want to check, how long I can run durable function so I created 3 functions

  1. Http function to kick start Orchestrator function
  2. Orchestrator function enter image description here
  3. Activity function

enter image description here

My DurableFunction is running and emitting logs in Application Insights from last 5 days and based on my code it would take 15 more days to complete.

I want to know that how to stop Orchestrator function manually?

I can see thousands of entry in ApplicationInsights requests table for single execution, Is there any way to check how many DurableFunction running in backend? and how much time taken by single execution?

I can see some information regarding orchestrator function in "DurableFunctionHubInstance" table but as MS recommended not rely on table.

2
azureazure-durable-functions
In addition to my answer below, I'd also like to suggest you look into the various patterns Durable Functions provides. The example you're showing here is function chaining and is sequential, you wait for the completion of the first function before executing the next one. In your case, you might use the fan-out/fan-in pattern instead so you can call activity functions in parallel. – Marc

2 Answers

12
votes

Since Durable Functions does a lot of checkpointing and replays the orchestration, normal logging might not always be very insightful.

Getting the status

There are several ways to query for the status of orchestrations. One of them is through the Azure Functions Core tools as George Chen mentioned.

Another way to query the status is by using the HTTP API of Durable Functions directly:

GET <rooturl>/runtime/webhooks/durableTask/instances?
    taskHub={taskHub}
    &connection={connectionName}
    &code={systemKey}
    &createdTimeFrom={timestamp}
    &createdTimeTo={timestamp}
    &runtimeStatus={runtimeStatus1,runtimeStatus2,...}
    &showInput=[true|false]
    &top={integer}

More info in the docs.

The HTTP API also has methods to purge orchestrations. Either a single one by ID or multiple by datetime/status.

DELETE <rooturl>/runtime/webhooks/durabletask/instances/{instanceId}
    ?taskHub={taskHub}
    &connection={connection}
    &code={systemKey}

Finally you can also manage your instances using the DurableOrchestrationClient API in C#. Here's a sample on GitHub: HttpGetStatusForMany.cs

I have written & vlogged about using the DurableOrchestrationClient API in case you want to know more about how to use this in C#.

Custom status

Small addition: it's possible to add a custom status object to the orchestration so you can add enriched information about the progress of the orchestration.

Getting the duration

When you query the status of an orchestration instance you get back a DurableOrchestrationStatus object. This contains two properties:

  • CreatedTime
  • LastUpdatedTime

I'm guessing you can subtract those and get a reasonable indication of the time it has taken.

3
votes

You could manage the Durable Functions orchestration instances with Azure Functions Core Tools.

Terminate instances: 

func durable terminate --id 0ab8c55a66644d68a3a8b220b12d209c --reason "It was time to be done."

Query instances with filters: you could add the parameter(runtime-status) to filter the running instances.

func durable get-instances --created-after 2018-03-10T13:57:31Z --created-before  2018-03-10T23:59Z --top 15

As for the time functions took, looks like it doesn't support. The similar parameter is the get-history.