Insert into RavenDB; Fastest way

Question

I want to import 100 million entries from a text file (each row is one csv-like entry) into a RavenDB database. What is the fastest way to do this?

Additional Notes:

I have not any indexes yet (I will create them after inserting the data). RavenDB is running in service mode on local machine with no security enhancements (yet; because I am still testing RavenDB). This test will run on 2 different machines, 1) 2 cores 4GB ram 2) 8 cores 12 GB ram.

I have done inserting a portion of this data (2 million entries) into RavenDB but it was not as fast as I would like. By using OpenAsyncSession and calling SaveChangesAsync for every 1024 records and again creating a new session by calling OpenAsyncSession and not waiting for return Task (returned by SaveChangesAsync) after 500`000 entries or so, I get an "Index out of range" exception that I can not root out. But if I wait for tasks to end (by creating them same as number of cores), process will succeed but not fast enough.

This code ran successfully:

using (var reader = new StreamReader(@"D:\*\DATA.TXT", Encoding.UTF8))
{
    string line = null;
    IAsyncDocumentSession session = null;

    var tasks = new List<Task>();
    var locCount = 0;

    while ((line = reader.ReadLine()) != null)
    {
        if (string.IsNullOrWhiteSpace(line)) continue;

        var loc = Parse(line);

        if (session == null) session = documentStore.OpenAsyncSession();

        session.Store(loc);
        locCount++;

        if (locCount % 1024 == 0 && session != null)
        {
            try
            {
                var t = session.SaveChangesAsync();
                tasks.Add(t);
                session = null;
            }
            catch (Exception x)
            {
                // ... something ...
            }
        }

        if (tasks.Count >= NUMBER_OF_CORES)
        {
            Task.WaitAll(tasks.ToArray());
            tasks.Clear();
        }
    }

    if (session != null)
    {
        if (tasks.Count > 0)
        {
            Task.WaitAll(tasks.ToArray());
            tasks.Clear();
        }
        session.SaveChangesAsync().Wait();
        session = null;
    }
}

Thanks

If you don't wait for completion, it will look fast of course. Because processing has not completed yet. This is not a valid benchmark. — usr
A document database is not about "records". You probably want to see how that data can be modeled to fit in a document database. For examples of similar scenarios see ayende.com/blog/154401/… and ayende.com/blog/156386/… — synhershko
@synhershko I know - to some level - it's about documents and I have used word "record" here because at the time I have not a better word for it; "entry" maybe? "object"? "information"? I do not know...Thanks! — Kaveh Shahbazian
My point - you may very well require doing an insert of a lot less than you think. The fastest way would be to use a simple console application to do this in batches - the session API will do that for you if you'll call SaveChanges on every bunch of them. Most of time you'll find reading the data from your sources will take the majority of time. — synhershko
@usr processing HAS been completed, RavenDB is ACID. Indexing may not have caught-up, but all the info is already available. — synhershko

Ayende Rahien Ayende Rahien · Accepted Answer · 2012-06-25T07:40:35

Kaveh,

There are a number of issues here.

1) RavenDB models very rarely map to CSV files. If you have a CSV file, you usually have tabular format, and that isn't a good format to port to RavenDB. You can probably get better results by getting good models.

2) You code, without the if (tasks.Count >= NUMBER_OF_CORES), will generate as many tasks as possible (subject to the limit of reading lines from the file, which is really fast. This will tend to generate thousands of concurrent tasks, and will overload the number of requests RavenDB can insert at once.

3) Use the standard session, use a batch size of 1,024 - 2,048. And just let it run. RavenDB is really good in optimizing, and I expect that you'll see thousands of inserts per second easily.

But, again, you are probably modeling things wrong.

Insert into RavenDB; Fastest way

1 Answers