SOLR 4.6.0 Indexing Concurrency Problem via CLI - org.apache.solr.common.SolrException: Index locked for write for core XXX

0
votes

I have a web application running in Tomcat with a SQL database and SOLR backend.

The task is implementing a standalone enhanced version of a webapp triggered functionality. Existing Webapp version involves querying a specific product from a SQL database (which also exist in SOLR) and re-indexing/updating it in a SOLR engine.

I am running a main class in a JAR file deployed via CLI as follows:

java -Dxxx.base=/usr/local/xxx -cp "xxxutils-4.12-SNAPSHOT.jar:/usr/local/tomcat/lib/servlet-api.jar:/usr/local/tomcat/conf:/usr/local/xxx/conf:*" org.sam.xxx.utils.SolrReIndexExec

My CLI version is slightly different to the web, it queries all products from the SQL database and tries to re-Index all the products.

My program is failing when it tries to add the documents to SOLR with the following error:

//About to add document to EmbeddedSolrServer  

Exception in thread "main" org.apache.solr.common.SolrException: SolrCore 'XXXX' is not available due to init failure: Index locked for write for core XXXX
        at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:818)
        at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:110)
        at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
        at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
        at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
        at org.test.sam.service.SolrService.addRecordToSolrCli(SolrService.java:270)
        at org.test.sam.service.SolrService.addProductToSolrCli(SolrService.java:517)
        at org.test.sam.function.ReIndexProductService.reIndexAllProducts(ReIndexProductService.java:106)
        at org.test.sam.utils.SolrReIndexExec.<init>(SolrReIndexExec.java:39)
        at org.test.sam.utils.SolrReIndexExec.main(SolrReIndexExec.java:52)
Caused by: org.apache.solr.common.SolrException: Index locked for write for core XXXX
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:834)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:625)
        at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:557)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:592)
        at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:271)
        at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:263)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.lucene.store.LockObtainFailedException: Index locked for write for core XXXX
        at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:491)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:755)
        ... 11 more

    -----------------------------------------------------------------------

After some investigation. I saw some post suggesting the following:

  1. Changing the lockType to simple in the solrConfig.xml, that potential fix made no difference.
  2. Deleting the write.lock file /usr/local/xxx/Data/solr/XXXX/data/index/write.lock and re-runing the JAR resulted in the error below:

Exception in thread "main" org.apache.solr.common.SolrException: SolrCore 'XXXX' is not available due to init failure: Error opening new searcher at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:818) at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:110) at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117) at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116) at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102) at org.test.sam.service.SolrService.addRecordToSolrCli(SolrService.java:270) at org.test.sam.service.SolrService.addProductToSolrCli(SolrService.java:517) at org.test.sam.function.ReIndexProductService.reIndexAllProducts(ReIndexProductService.java:106) at org.test.sam.utils.SolrReIndexExec.(SolrReIndexExec.java:39) at org.test.sam.utils.SolrReIndexExec.main(SolrReIndexExec.java:52) Caused by: org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.(SolrCore.java:834) at org.apache.solr.core.SolrCore.(SolrCore.java:625) at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:557) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:592) at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:271) at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:263) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1477) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1589) at org.apache.solr.core.SolrCore.(SolrCore.java:821) ... 11 more Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/usr/local/pm6/Data/solr/XXXX/data/index/write.lock at org.apache.lucene.store.Lock.obtain(Lock.java:84) at org.apache.lucene.index.IndexWriter.(IndexWriter.java:702) at org.apache.solr.update.SolrIndexWriter.(SolrIndexWriter.java:77) at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:64) at org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:267) at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:110) at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1440) ... 13 more

Our SOLR setup is a single SOLR Core per instance per machine. I think the root cause is that the SOLR instance is initialized via the webapp and I am trying to access it via CLI. Ideally, I would like to use the same instance created by the webapp.

Below is my code that create the solr server via CLI:

//Gets the solr server. 
public synchronized static EmbeddedSolrServer getSolrServerCli() {
    String basePath = "/usr/local/xxx/Data/solr";
    CoreContainer container = new CoreContainer(basePath);
    container.load();
    return new EmbeddedSolrServer(container, "XXXX");
}



//Then add document

 EmbeddedSolrServer server = SolrService.getSolrServerCli();

 UpdateResponse x = server.add(doc);

Any ideas will be appreciated. Thanks

1
javaconcurrencysolrlucene
Any reason why you don't make the request to the webapp instead of creating an embedded solr server? The embedded solr server is meant to be used to have a Solr instance inside an application, not for updating another server. – MatsLindh
Hello @MatsLindh thanks for your comment. We wanted to run the reIndexing adhoc-ly on the service, hence the CLI approach. Please, can you shed some light on "The embedded solr server is meant to be used to have a Solr instance inside an application, not for updating another server."? – Mega
If you were building a custom application that wants to have Solr's capabilities inside your own application (for example a desktop application), you'd use the EmbeddedSolrServer. It's also useful for integration tests and similar use cases. It's not meant for modifying an existing index, which the application that owns the index (in this case your webapp under Tomcat) has locked for modification. If another application were to change the index beneat the Tomcat server, it would have no idea about what had changed and no way to know if it had been done in a stable and secure way. – MatsLindh
Instead, send the update request to the server through SolrJ - that is the preferred way to interact with Solr. Unless you have some very specific reason to not access the Solr server through the HTTP API, do that. Making stuff work without going through the preferred update mechanism will require you to jump through multiple hoops and make sure everything is done in a safe manner. – MatsLindh
Wow, that makes sense, thank you. I was going to use HttpClient to invoke the existing servlet which does single document re-indexing. Then clone servlet and tweak to re-index multiple docs. Please, do you know any good resource or reference for the SolrJ for my use case. I will also google, but you might have one in mind. Our SOLR version is 4.6.0. – Mega

1 Answers

0
votes

My solution was using an existing web application entry point to interact with SOLR Index. I've implemented a standalone application triggered via CLI, which calls a servlet via a rest client. The servlet in turn uses a service class to interact with SOLR Index in a servlet context. Hence concurrency issue was avoided.

See my post here for more details.

How to Improve my Java App that migrates millions of records from MariaDB database to a SOLR Server

@MatsLindh - was my design buddy. Thanks Pal.