0
votes

I have a structure of nested Parallel.For and PLINQ statements in my small console app that basically performs network-bound operation(performing http requests) like following:

a list of users is filled from DB where then I do following:

Parallel.For(0,users.count(), index=>{

// here I try to perform HTTP requests for multiple users

});

Then inside this for loop I perform a plinq statement for fetching this user's info via HTTP requests.

So that now in total I get two nested loops like following:

Parallel.For(0,users.count(), index=>{

// Some stuff is done before the PLINQ statement is called... 
newFilteredList.AsParallel().WithDegreeOfParallelism(60).ForAll(qqmethod =>
    {
        var xdocic = new XmlDocument();
        xdocic.LoadXml(SendXMLRequestToEbay(null, null, qqmethod.ItemID, true, TotalDaysSinceLastUpdate.ToString(), null));
        int TotalPages = 0;
        if (xdocic.GetElementsByTagName("TotalNumberOfPages").Item(0) != null)
        {
            TotalPages = Convert.ToInt32(xdocic.GetElementsByTagName("TotalNumberOfPages").Item(0).InnerText);
        }
        if (TotalPages > 1)
        {
            for (int i = 1; i < TotalPages + 1; i++)
            {
                Products.Add(SendXMLRequestToEbay(null, null, qqmethod.ItemID, false, TotalDaysSinceLastUpdate.ToString(), i.ToString()));
            }
        }
        else
        {
            Products.Add(SendXMLRequestToEbay(null, null, qqmethod.ItemID, false, TotalDaysSinceLastUpdate.ToString(), "1"));
        }
    });
});

I tried using the outer for loop just as a regular one ,and I noticed that it was performing much faster and better than like this.

What worries me here mostly is that I was checking the utilization of CPU when running the console app like this, it's always nearby 0.5-3% of total CPU power...

So the way I'm trying to perform HTTP requests is like this:

15 users at a time * amount of HTTP requests for those 15 users.

What am I doing wrong here?

The Parallel classes are not for non CPU bound work. HTTP requests are I/O bound. Also what is the type of Products, if it is just a List<T> that class is not thread safe and you are breaking the class while you write to it with multiple threads concurrently.Scott Chamberlain
@ScottChamberlain What would you suggest me that I replace the first outer for loop with then ? I'd like to be able to process 15 and send 15 simultaneous http requests for those users, how could I do that ?User987
@ScottChamberlain the Products is concurrent bag =)User987
Use Dataflow instead, you can have two blocks in a chain, one for sending the request, one for processing, you can set the max degree of parallelism for each block.Scott Chamberlain
See the 2nd half of this old answer of mine, it has two blocks, one that gets records in parallel then one that processes those records one at a time to add it to a list. you could transform the 2nd block to do your "processing" you need and set it to a higher degree of parallel than 1.Scott Chamberlain