Azure Durable Function getting slower over time due to storage

Question

I have to fetch update for 20000 records from external API daily.When function starts it perform well, but after some time(1hr) it become slow due to data generated in storage. Can I clear instance history while running the function. or any other way to achieve the same thing.

    [FunctionName("FanInOut")]
    public async Task Run([OrchestrationTrigger] IDurableOrchestrationContext context)
    {
        var parallelTasks = new List<Task<int>>();
        var dataList = new List<StockData>() { };// datalist as list of item from DB

        // Get a list of N work items to process in parallel.
        var numberOfBatches = dataList.Count / 5;
        for (int i = 0; i < numberOfBatches; i++)
        {
            var dataToProcess = dataList.Skip(0).Take(5).ToArray();// processing in batch of 5
            for (int j = 0; j < dataToProcess.Count(); i++)
            {
                Task<int> task = context.CallActivityAsync<int>("F2", dataToProcess[i]);
                parallelTasks.Add(task);
            }
            await Task.WhenAll(parallelTasks);   }}}

Thiago Custodio Thiago Custodio · Accepted Answer · 2020-11-11T14:43:52

Deleting history will put Durable Functions on an invalid state as it uses storage data as checkpoints.

I believe you should use Fan IN / Fan Out and divide the process to multiple tasks or even add an initial step that would trigger your orchestrator multiple times (e.g. pass 1000 items at a time)

Azure Durable Function getting slower over time due to storage

1 Answers