SolrCloud replicas locking related error

Question

We have three VMs: on the first two systems we installed Tomcat 6, Zookeeper 3.4.6 and Solr 4.10. On the third node wi installed TOmcat 6 and Zookeeper 3.4.6 So we have a two nodes solrcloud cluster and a three nodes zookeeper cluster.

The first two VMs are srvmsolr01 and srvmsolr02.

We can create collections and shards without errors. We are not able to add replicas.

For example the following command:

<i>http://srvmsolr01.test.com:8080/solr/admin/collections?action=CREATE&name=mycollection3&numShards=2&replicationFactor=2&maxShardsPerNode=2</i>

give us the error:

<i>Caused by: org.apache.solr.common.SolrException: Unable to create core
[shard2_replica2]
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:507)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:466)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:575)
        ... 18 more
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:873)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:646)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
        ... 20 more
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
        at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1565)
        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1677)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:845)
        ... 22 more
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock
obtain timed out: NativeFSLock@/app/zookeeper/data/index/write.lock
        at org.apache.lucene.store.Lock.obtain(Lock.java:89)
        at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:755)
        at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:77)
        at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:64)
        at org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:279)
        at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:111)
        at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1528)
        ... 24 more </i>

The same happens if we create a collection with two shards and then we try to add each replica individually:

<i>http://srvmsolr02.test.com:8080/solr/admin/cores?action=CREATE&name=shard1_replica2&collection=collection1&shard=shard1
http://srvmsolr01.test.com:8080/solr/admin/cores?action=CREATE&name=shard2_replica1&collection=collection1&shard=shard2</i>

We double checked the configuration but we are not able to understand how to achieve the our goal: two solrcloud node with a collection made of two shard and two replicas, one for each shard

Regards Giova

Did you ever figure this out? I'm experiencing a similar problem that appears to be locking related. I tried all the locking answers I found, but nothing helped. — LandonC

Giova Giova · Accepted Answer · 2015-07-16T08:33:31

I found the problem. The error was raised because of the datadir parameter in SolrConfig.xml With this parameter Solr was creating all cores in the same directory and each core has a lock file, so only one core were locked in a specific moment.

Regards Giova

SolrCloud replicas locking related error

1 Answers