(C#) I have an Http Starter function which triggers an Orchestration function containing five asynchronous Activity functions within a While loop.
Pseudo-structure of ideal functional layout which I'm trying to achieve:
static long latestLogEventIdx = new long;
static long batchIncrement = 1000;
[FunctionName("MainOrchestrator")]
public static async Task<List<string>> RunMainOrchestrator([context])
await context.CallActivityAsync<string>("ActivityFunc1", data)
while (latestLogEventIdx < maxIndex)
{
try
{
await context.CallActivityAsync<string>("ActivityFunc2", data)
await context.CallActivityAsync<string>("ActivityFunc3", data)
await context.CallActivityAsync<string>("ActivityFunc4", data)
await context.CallActivityAsync<string>("ActivityFunc5", data)
}
catch
{
err
}
}
latestLogEventIdx += batchIncrement
return outputs
}
(The only difference from the above code to my actual code is that my actual code has all activity functions within the while loop, but ideally the first would only execute once. I'm just not quite sure how to achieve that).
The Activity functions make queries to several SQL databases and process the response data. The script is long-running and consequently needs to process the data in batches between record indexes to avoid timing out and other issues, which is what I thought to use my While loop for - to iterate over the Activity functions while within a specified range of indices returned from my first Activity function.
I created a global variable to keep track of the latest index logged to a SQL database (the first activity function gets and sets this index). Then I query another database for a range of data between the latest index logged + batchIncrement which I store in a global list (second activity function).
Then I process this data in the third Activity function using the data I stored in my global var from the previous activity function. Finally, I save it to another DB in the last activity function.
Then I shift the latestLogEventIdx forward by the batchIncrement and start over, ideally from the second Activity function (if somehow I can skip the first Activity function).
This is causing some issues beyond the first pass which now seems logical based on my little understanding of Durable functions and the execution order of Orchestration functions which I understand (minimally) from this MS Visual Studio tutorial on YouTube.
It seems, based on the log.LogInformation output in my terminal, as if the Orchestrator is being called on top of itself after the first full execution; executing Activity functions on top of one another and in random orders, as the output prints several consecutive executions of each function in seemingly random orders only becoming more sporadic. These errors occur after the first full pass of the Orchestrator function.
Any help on how to achieve a working approach of the semi-functional pseudo-code above would be much appreciated!