Insert 14 billion records in Azure Table Storage

4
votes

In one of my project I received customer order details at the middle of each month which is an about 14 billion lines file. I need to upload them into my system (1 line per record) within 1 week then users can query.

I decided to use table storage to store based on price and performance consideration. But I found the performance of table storage is "2000 entities per second per partition" and "20,000 entities per second per account". https://azure.microsoft.com/en-us/documentation/articles/storage-scalability-targets/

This means if I was using 1 storage account I need about 1 month to upload them which is not acceptable.

Is there any solution I can speed up to finish the upload task within 1 week?

1
azureazure-table-storage
You might want to look into DynamoDB instead of Azure - Androme
2 questions: 1) How big is 1 record in size (1KB, 2KB etc.)? 2) How would the users query this data? Is it simple order lookup by Id? - Gaurav Mantri
@DoomStone My app was built on top of azure so I have to use it. - Shaun Xu
@GauravMantri 1) Less than 1 KB I think. 2) Each record contains customer id and order details. User will query based on their customer id. So what I did is to use customer id as the partition key and assign each row a GUID as row key. - Shaun Xu
@ShaunXu why? you can host apps on Azure that uses Dynamo DB, Amazon also have a nice C# implementation. - Androme

1 Answers

2
votes

The simple answer to this is to use multiple storage accounts. If you partition the data and stripe it across multiple storage accounts you can get as much performance as you need from it. You just need another layer to aggregate the data afterwards.

You could potentially have a slower process that is creating one large master table in the background.

You may have found this already, but there is an excellent article about importing large datasets into Azure Tables