Multi-Threaded Component Design Question

0
votes

We have 10 denormalized tables that contains large amounts of data (11,000,000) in many of them. The client wants the user to be able to download up to 100K of records at a time. What I am wondering is the design of a background process to handle this. I was thinking of possibly setting up a queue for each of the tables and having multiple threads handling the requests. I'm definetly not an expert on designing multi-threaded components. Do you guys think I'm on the right track here? Any suggestions on implementation?

Thanks!

3
c#.netasp.netmultithreading
When you say 'download up to 100K of records at a time', is that to export (or something similar where all the data is needed)? Or to view on screen (where you could paginate and give the illusion of all the data being there)? - Iain Ward
What would the queue be doing? - Marcelo Cantos
Correct, an export to excel (or csv). Users will have a web interface, so I would need to drop a request into some type of a queue. The queue would simply store the report type and search params. - Marco
And yes, we will be utilizing data paging for the front end, but they need the entire set when working offline in excel. - Marco
@Marco - Just to clarify the use case, The user makes the request on your asp.net page - one way probably - you will process the request at the server and then once processed, present the user with a link (?) to an xls / csv file on the server with 100K records, which the user can download and view? Is that correct? - Jagmag

3 Answers

1
votes

Design scenario

1- Since this is an Asp.Net applicaion , how many concurrent users will download the data?

2- Are you thinking to give the download data facility with in your application , did you thought about process re-cycling?

3- What if the user session expires but download is still in progress?

4- You probably moving towards a streaming server.

i think , you should think about these scenarion first than decide which type of application is actually required.

in my thoughts

i think , you should think of creating a windows based solution which will be downloaded to the client computer and made use of client resources (threads etc) to connect to the server and your server must support streaming.

0
votes

Will the users be making custom queries or just downloading "chunks" of the table?

If the latter then why not just generate the CSV for each chunk ahead-of-time? You'll only need 110 chunks if each chunk has 100,000 records and the table only has 11,000,000 records in total.

0
votes

I think this is one way of doing it.

You can call another page and put the process in next page.

Following article might be useful.

http://codebetter.com/blogs/peter.van.ooijen/archive/2006/06/20/146582.aspx