C# Make async http requests within a foreach loop

Question

I need to make multiple webrequests where URIs are in a DataTable. Earlier I had the below code. But I realized this makes synchronous calls as await would wait till GET/POST call is complete and response is processed then it proceeds to next iteration.

foreach (DataRow dr in dt.Rows)
{
    activeTasks.Add(SendRequestAsync(dr));
    Task.WhenAll(activeTasks).Wait();
}

private async Task<string> SendRequestAsync(DataRow dr)
{
    using (var client = new HttpClient())
    {
        string reqMethod = (dr["RequestMethod"] != null && dr["RequestMethod"].ToString() != "") ? dr["RequestMethod"].ToString() : "GET";
        client.BaseAddress = new Uri(dr["URL"].ToString());
        client.DefaultRequestHeaders.Accept.Clear();
        string reqContentType = (dr["RequestContentType"] != null && dr["RequestContentType"].ToString() != "") ? dr["RequestContentType"].ToString() : "text/xml";
        client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue(reqContentType));

        HttpResponseMessage response = null;
        try
        {
            if (reqMethod == "GET")
                response = await client.GetAsync(client.BaseAddress.AbsoluteUri);
            else
                response = await client.PostAsync(client.BaseAddress.AbsoluteUri, null);

            response.EnsureSuccessStatusCode();
            var responseText = await response.Content.ReadAsStringAsync();
            return responseText;
        }
        catch (Exception e)
        {
            return "-1";
        }
    }
}

Then I came across Parallel feature and used Parallel.ForEach instead. Like this:

Parallel.ForEach(rows, dr =>
{
    activeTasks.Add(SendRequestAsync(dr));
    Task.WhenAll(activeTasks).Wait();
});

This works fine, parallelism is achieved, requests are asynchronous and it completes within fraction of a time as compared to earlier solution. But the problem is it is not reliable - at times I get errors like

System.IndexOutOfRangeException: Index was outside the bounds of the array
System.InvalidOperationException: Collection was modified; enumeration operation may not execute.

Is there anyway we can achieve http async calls within a foreach?

I wouldn't use the Parallel.ForEach for this. Really you just want to move your Task.WhenAll outside of the loop that you are adding tasks to your task list in. — Jonathon Chase
When using parallel foreach you want to make sure you use threadsafe collections from the concurrent collections namespace docs.microsoft.com/en-us/dotnet/api/… — DetectivePikachu
As an aside, if you do decide to make your calls concurrently, you may need to increase the timeout on each call, as you may have a request blocked by the maximum number of simultaneous sockets being open in some environments. — Jonathon Chase
From the documentation: HttpClient is intended to be instantiated once and re-used throughout the life of an application. Instantiating an HttpClient class for every request will exhaust the number of sockets available under heavy loads. — Theodor Zoulias

Kevin Anderson Kevin Anderson · Accepted Answer · 2019-10-22T15:41:02

As @Johnathon_Chase said, just move your WhenAll() call outside of the loop:

foreach (DataRow dr in dt.Rows)
{
    activeTasks.Add(SendRequestAsync(dr));
}
Task.WhenAll(activeTasks).Wait();

The for loop populates the collection, and then Task.WhenAll() blocks while the requests complete.

C# Make async http requests within a foreach loop

2 Answers