How can I limit Parallel.ForEach?

333
votes

I have a Parallel.ForEach() async loop with which I download some webpages. My bandwidth is limited so I can download only x pages per time but Parallel.ForEach executes whole list of desired webpages.

Is there a way to limit thread number or any other limiter while running Parallel.ForEach?

Demo code:

Parallel.ForEach(listOfWebpages, webpage => {
  Download(webpage);
});

The real task has nothing to do with webpages, so creative web crawling solutions won't help.

4
c#.netasynchronousparallel.foreach
@jKlaus If the list isn't modified e.g. it's just a set of URLs, I can't really see the issue? - Shiv
@Shiv, given enough time you will... Count your number of executions and compare it to the count of the list. - jKlaus
@jKlaus What are you saying will go wrong? - Shiv
@jKlaus you are modifying a non-threadsafe element (the integer). I would expect it to not work in that scenario. The OP on the other hand is not modifying anything that needs to be threadsafe. - Shiv
@jKlaus Here is an example of Parallel.ForEach that sets the count correctly > dotnetfiddle.net/moqP2C. MSDN Link: msdn.microsoft.com/en-us/library/dd997393(v=vs.110).aspx - jhamm

4 Answers

618
votes

You can specify a MaxDegreeOfParallelism in a ParallelOptions parameter:

Parallel.ForEach(
    listOfWebpages,
    new ParallelOptions { MaxDegreeOfParallelism = 4 },
    webpage => { Download(webpage); }
);

MSDN: Parallel.ForEach

MSDN: ParallelOptions.MaxDegreeOfParallelism

46
votes

You can use ParallelOptions and set MaxDegreeOfParallelism to limit the number of concurrent threads:

Parallel.ForEach(
    listOfwebpages, 
    new ParallelOptions{MaxDegreeOfParallelism=2}, 
    webpage => {Download(webpage);});
23
votes

Use another overload of Parallel.Foreach that takes a ParallelOptions instance, and set MaxDegreeOfParallelism to limit how many instances execute in parallel.

15
votes

And for the VB.net users (syntax is weird and difficult to find)...

Parallel.ForEach(listOfWebpages, New ParallelOptions() With {.MaxDegreeOfParallelism = 8}, Sub(webpage)
......end sub)