How can I asynchronously get data from many websites, and add them to a collection?

Question

I am trying to call many different web services at the same time and aggregating the data.

My intention is to create a Task for each web call, pass a shared container to each task, and store data from each call in the container. As long as I can get data from each web call into the shared container, I am happy.

I have created an example of what I'm trying to do - however, it sometimes crashes with an exception on the Task.WaitAll line: "One or more errors occurred. (Source array was not long enough. Check the source index, length, and the array's lower bounds. Parameter name: sourceArray)".

I am new to using async/await and multithreading.

using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Threading.Tasks;
using System.Linq;

namespace ConsoleApp1
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Starting tasks...");
            List<Task> tasks = new List<Task>();
            List<char> container = new List<char>();
            for (int i = 0; i < 80; i++)
            {
                tasks.Add(LongTask(container));
            }
            Task.WaitAll(tasks.ToArray());
            Console.WriteLine("Checkpoint 1.");
            Console.WriteLine("Tasks Finished");

            Console.ReadLine();
        }

        public static async Task<string> LongTask(List<char> container)
        {
            var client = new HttpClient();
            var text = await client.GetAsync("http://www.google.com");

            var myList = text.StatusCode.ToString().ToList();

            container.AddRange(myList);

            return text.StatusCode.ToString();
        }
    }
}

List<char>.AddRange( ) is not thread-safe. Have you tried using something like ConcurrentBag? — Dylan Nicholson
Possible duplicate of multiple awaits vs Task.WaitAll - equivalent? — GolezTrol

Pranay Rana Pranay Rana · Accepted Answer · 2017-12-13T08:59:48

you can make use of lock construct around current list when adding data that will also work.

below will go at class level

private static object _lock = new object();

this will go in your method

var myList = text.StatusCode.ToString().ToList();
lock(_lock)
{
  container.AddRange(myList);
}

or You can make use of ConcurrentBag

so in you code replace List by this , in main method

ConcurrentBag<string> container = new ConcurrentBag<string>();

and in the LongTask( method do as below.

 var myList = text.StatusCode.ToString().ToList();               
 container.AddRange(myList);

here I am suggesting you list of string as ConcurrentBag doenst have AddRange method and if you do write foreach to add in ConcurrentBag than might be chance other thread also add char in between , instead of that once all get completed you can take string out of ConcurrentBag and convert it into char array or list.

Note: ConcurrentBag<T> is a thread-safe bag implementation, optimized for scenarios where the same thread will be both producing and consuming data stored in the bag.

How can I asynchronously get data from many websites, and add them to a collection?

1 Answers