When the document size is larger than memory in Solr?

Question

I have 3gb document size and 1.5gb ram. When I start an instance of Solr, it automatically starts to consume 1gb memory. And then I enter the curl command that updates document from console. After a few seconds the Solr instance fill all memory and starts to consume cpu. The update process never finish and cpu usage increases in time. Is there any way to handle this problem?

My Solr schema is in here: https://github.com/buremba/mbslave/tree/master/solr

Answer 1

How many documents do you have? Or is this one 3gb document?

If you have more than one doc, you can add them in batches and commit after each batch. Each batch could be less than 100mb. In that way you can manage the heap size.

Also, to get around the immediate issue, you can increase the os virtual memory and use higher java heap size (-Xmx of 4gb).