How to properly detect when multiple threads within another thread are fully complete?

Question

I'm running several BackgroundWorkerthreads that are being used to execute queries to retrieve DataSets all within another BackgroundWorker thread. Let's call the thread that is running these multiple threads the 'Host Thread' and the others 'Query Thread'. What I am trying to do is to tell when all of the query threads are finished populating their DataSets by utilizing the host thread's RunWorkerCompleted event. The first line in this event handler is

while (dataSets.Count < count) { Thread.Sleep(100); } //dataSets is a Dictionary<string, DataSet>

where count is the total amount of DataSets that are expected to be returned. My issue seems to be that dataSets.Count seems to become == to count before all of the DataSets are populated.

Here is my full code (Unhelpful/Sensitive information removed)

        var hostThread = new BackgroundWorker();
        hostThread.RunWorkerCompleted += new RunWorkerCompletedEventHandler(queryWorker_RunWorkerCompleted);
        hostThread.DoWork += (send, even) =>
            {
                foreach (var cs in _connectionStrings)
                {
                    var queryThread = new BackgroundWorker();
                    queryThread.DoWork += (se, eve) =>
                    {
                        var set = DataHandlers.TryGetDataSet(_query, cs, domain, username, pass);
                        dataSets.Add(((DataRow)set.Tables[0].Rows[0]).ItemArray[0].ToString(), set);
                    };
                    queryThread.RunWorkerAsync();
                }
            };
        hostThread.RunWorkerAsync();

The RunWorkerCompleted :

 var bw = new BackgroundWorker();
        bw.DoWork += (s, ev) =>
            {
                //Waiting for all DataSets to get populated
                while (dataSets.Count < count) { Thread.Sleep(100); }
                //Thread.Sleep(5000); If I add this, everything works fine, but when I start running more queries in each query thread this needs to be increased.
                this.Invoke((MethodInvoker)delegate()
                {
                    this.Cursor = Cursors.Default;
                    this.Hide();
                    foreach (var set in dataSets)
                    {
                            if (set == null)
                                break;
                            //THIS BLOCK IS NEVER HIT IF I LEAVE OUT THE FIVE SECOND SLEEP
                            var workflowList = new List<string>();
                            foreach (var row in set.Value.Tables[0].Rows)
                            {
                                workflowList.Add(((DataRow)row).ItemArray[_licensed ? 1 : 0].ToString());
                            }
                            ((MainForm)this.OwnedForms[0]).ClientWorkflows = new KeyValuePair<string, List<string>>(set.Key, workflowList);
                    }
                    //This gets hit before setting properties on a child form because it still thinks there are no DataSets in the dataSets dictionary
                    ((MainForm)this.OwnedForms[0]).ShowDialog();
                    this.Close();
                });

            };
        bw.RunWorkerAsync();

So as I stated in the comments in the code - I know that at some point the DataSets will be valid as long as I add a long enough sleep after the while loop. So what would be the best way to tell when all of the query threads are actually completed within the host thread completed event handler?

EDIT: Per @ndd this is what I ended up using.

  var queryTasks = new List<Task>();

        var parentTask = Task.Factory.StartNew(() =>
            {
                foreach (var cs in appConfigStrings)
                {
                    queryTasks.Add(Task.Factory.StartNew(() => GetDataSets(mainForm, cs.Key, cs.Value)));
                }
                var array = queryTasks.ToArray();
                Task.WaitAll(array);
            });

        parentTask.ContinueWith((t) =>
            {
                this.Invoke((MethodInvoker)delegate()
               {
                   this.Cursor = Cursors.Default;
                   this.Hide();
                   foreach (var set in dataSets)
                   {
                       var workflowList = new List<string>();
                       foreach (var row in set.Value.Tables[0].Rows)
                       {
                           workflowList.Add(((DataRow)row).ItemArray[_licensed ? 1 : 0].ToString());
                       }
                       ((MainForm)this.OwnedForms[0]).ClientWorkflows = new KeyValuePair<string, List<string>>(set.Key, workflowList);
                   }
                   ((MainForm)this.OwnedForms[0]).ShowDialog();
                   this.Close();
               });
            });

Wow, I have yet to be introduced to task parallelism. Just from glancing at it for a few moments I can tell this will definitely be my best option. I'm still perplexed as to why my solution fails to work. EDIT: Well actually I'm restricted to .net fw 4.0. Damn! @ndd — Jonathan Carroll
You can't just wait some amount of time, that might take too long, and if one of the BackgroundWorkers experiences an exception it will never return its data set. You need a way for each of your query threads to tell the host thread that they're done. One way would be to use an AutoResetEvent for each Query Thread that it sets when its finished and have the Host Thread wait on all of them. I don't know the TPL either, so this is the obvious way to me. — Tony Vitabile

ndd ndd · Accepted Answer · 2015-10-30T17:12:44

Personally I am in never favor of Sleep as it is not predictable. If I had to use BackgroundWorker then I would likely go with IsBusy property to determine whether the BackgroundThread is done or not.

Sample code with TPL, please note this is just an example, in real world you may want to handle exceptions, pass cancellation token and other things :)

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace BGToTPL
{
    class Program
    {
        static void Main(string[] args)
        {
            Task[] tasks = new Task[20];
            //Parent task is starting 20 child tasks
            var parentTask = Task.Run(() =>
            {
                Console.WriteLine("Parent threadid: " + System.Threading.Thread.CurrentThread.ManagedThreadId);
                for (int i = 0; i < 20; i++)
                {
                    tasks[i] = Task.Factory.StartNew(() =>
                    {
                        Console.WriteLine("Child threadid: " + System.Threading.Thread.CurrentThread.ManagedThreadId);                      
                        Task.Delay(15000);
                    });
                }
            });

            parentTask.Wait();

            Console.WriteLine("Parent task has started creating and running all the child tasks, now waiting for child tasks to be over.");

            //Now wait for all the tasks to be done
            Task.WaitAll(tasks);

            Console.WriteLine("All the tasks are done");
            Console.ReadKey();
        }
    }
}

And the output

How to properly detect when multiple threads within another thread are fully complete?

2 Answers