How to Prevent Solr from doing commit on restart that deletes the index

Question

**update From what I can tell, the delete and rebuild commits done on the main machine get cached and then when the index is copied to another machine and solr is restarted it somehow finds those cached commits somewhere and runs them on startup. This is bad for us because solr is hosed while it's redoing those cached commits.

How to disable that?

**original

We've recently changed from Solr 3 to Solr 8.

The index is built on one machine, then copied to 4 other machines.

It is a 'static' list of products. Once a week or so we delete and rebuild the index on machine 1 and copy to the other machines, then restart solr on those machines. The index is not modified (add or delete) in between rebuilds.

With Solr 3, when we restarted Solr after copying the new index, everything would work as we expected. It would simply just use the index we copied over.

Now with Solr 8, we are finding that restarting the other machines causes DirectUpdateHandler2/LogUpdateProcessorFactory to "REMOVE ALL DOCUMENTS FROM INDEX", do a commit and send a delete query.

This never happens on the machine the index is built on, only our slave machines that the index gets copied to.

We do not feel like replication is necessary in our case since the index remains static, and it's simpler for us to rsync the indexes to the other machines.

I'm having trouble finding what needs to change in solrconfig.xml to prevent it from sending that deleteByQuery on startup. Seems to only do it if enough time has passed since we last rebuilt the index.

This is empty in the config:

<updateHandler class="solr.DirectUpdateHandler2">
</updateHandler>

**this did not make a difference I just added this hoping it's going to make a difference:

    <deletionPolicy class="solr.SolrDeletionPolicy">

    <!-- The number of commit points to be kept -->
     <str name="maxCommitsToKeep">1</str>
    <!-- The number of optimized commit points to be kept -->
     <str name="maxOptimizedCommitsToKeep">0</str> 
    <!--
        Delete all commit points once they have reached the given age.
        Supports DateMathParser syntax e.g.
      -->
    <!--
       <str name="maxCommitAge">30MINUTES</str>
       <str name="maxCommitAge">1DAY</str>
    -->

    </deletionPolicy>

Solr is ran local and not in cloud mode. I do not use the /bin/solr file to start/stop. I have an rc.d that starts it.

Here are the relevant solr logs. Looks like it does a delete query and then adds them back but the index folder stays empty?

INFO  (searcherExecutor-11-thread-1-processing-x:core0) [   x:core0] o.a.s.c.SolrCore [core0] Registered new searcher Searcher@42dff6af[core0] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_6j(8.0.0):C26532:[diagnostics={java.runtime.version=1.8.0_202-b08, java.vendor=Oracle Corporation, java.version=1.8.0_202, java.vm.version=25.202-b08, lucene.version=8.0.0, os=FreeBSD, os.arch=amd64, os.version=12.0-RELEASE-p3, source=flush, timestamp=1557748786052}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]) Uninverting(_6k(8.0.0):C27135:[diagnostics={java.runtime.version=1.8.0_202-b08, java.vendor=Oracle Corporation, java.version=1.8.0_202, java.vm.version=25.202-b08, lucene.version=8.0.0, os=FreeBSD, os.arch=amd64, os.version=12.0-RELEASE-p3, source=flush, timestamp=1557748821980}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]) Uninverting(_6l(8.0.0):C25295:[diagnostics={java.runtime.version=1.8.0_202-b08, java.vendor=Oracle Corporation, java.version=1.8.0_202, java.vm.version=25.202-b08, lucene.version=8.0.0, os=FreeBSD, os.arch=amd64, os.version=12.0-RELEASE-p3, source=flush, timestamp=1557748857211}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]) Uninverting(_6m(8.0.0):C18995:[diagnostics={java.runtime.version=1.8.0_202-b08, java.vendor=Oracle Corporation, java.version=1.8.0_202, java.vm.version=25.202-b08, lucene.version=8.0.0, os=FreeBSD, os.arch=amd64, os.version=12.0-RELEASE-p3, source=flush, timestamp=1557748884975}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}])))}
INFO  (qtp349420578-20) [   x:core0] o.a.s.u.DirectUpdateHandler2 [core0] REMOVING ALL DOCUMENTS FROM INDEX
INFO  (qtp349420578-20) [   x:core0] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
INFO  (qtp349420578-20) [   x:core0] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@29a2d524 commitCommandVersion:0
INFO  (qtp349420578-20) [   x:core0] o.a.s.s.SolrIndexSearcher Opening [Searcher@2613a525[core0] main]
INFO  (qtp349420578-20) [   x:core0] o.a.s.u.DirectUpdateHandler2 end_commit_flush
INFO  (searcherExecutor-11-thread-1-processing-x:core0) [   x:core0] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@2613a525[core0] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
INFO  (searcherExecutor-11-thread-1-processing-x:core0) [   x:core0] o.a.s.c.QuerySenderListener QuerySenderListener done.
INFO  (searcherExecutor-11-thread-1-processing-x:core0) [   x:core0] o.a.s.c.SolrCore [core0] Registered new searcher Searcher@2613a525[core0] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
INFO  (qtp349420578-20) [   x:core0] o.a.s.u.p.LogUpdateProcessorFactory [core0]  webapp=/solr path=/update params={commit=true}{deleteByQuery=*:*,commit=} 0 145
INFO  (qtp349420578-14) [   x:core1] o.a.s.u.DirectUpdateHandler2 [core1] REMOVING ALL DOCUMENTS FROM INDEX
INFO  (qtp349420578-14) [   x:core1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
INFO  (qtp349420578-14) [   x:core1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@42e13429 commitCommandVersion:0
INFO  (qtp349420578-14) [   x:core1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3dc3e12[core1] main]
INFO  (qtp349420578-14) [   x:core1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
INFO  (searcherExecutor-10-thread-1-processing-x:core1) [   x:core1] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@3dc3e12[core1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
INFO  (searcherExecutor-10-thread-1-processing-x:core1) [   x:core1] o.a.s.c.QuerySenderListener QuerySenderListener done.
INFO  (searcherExecutor-10-thread-1-processing-x:core1) [   x:core1] o.a.s.c.SolrCore [core1] Registered new searcher Searcher@3dc3e12[core1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
INFO  (qtp349420578-14) [   x:core1] o.a.s.u.p.LogUpdateProcessorFactory [core1]  webapp=/solr path=/update params={commit=true}{deleteByQuery=*:*,commit=} 0 24
INFO  (qtp349420578-16) [   x:core0] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
INFO  (qtp349420578-16) [   x:core0] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@29a2d524 commitCommandVersion:0
INFO  (qtp349420578-16) [   x:core0] o.a.s.s.SolrIndexSearcher Opening [Searcher@8c5c9e1[core0] main]
INFO  (qtp349420578-16) [   x:core0] o.a.s.u.DirectUpdateHandler2 end_commit_flush
INFO  (searcherExecutor-11-thread-1-processing-x:core0) [   x:core0] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@8c5c9e1[core0] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_70(8.0.0):C60051:[diagnostics={os=FreeBSD, java.vendor=Oracle Corporation, java.version=1.8.0_202, java.vm.version=25.202-b08, lucene.version=8.0.0, mergeMaxNumSegments=-1, os.arch=amd64, java.runtime.version=1.8.0_202-b08, source=merge, mergeFactor=10, os.version=12.0-RELEASE-p3, timestamp=1557766812128}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]) Uninverting(_6o(8.0.0):C6151:[diagnostics={os=FreeBSD, java.vendor=Oracle Corporation, java.version=1.8.0_202, java.vm.version=25.202-b08, lucene.version=8.0.0, os.arch=amd64, java.runtime.version=1.8.0_202-b08, source=flush, os.version=12.0-RELEASE-p3, timestamp=1557766730928}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]) Uninverting(_6q(8.0.0):C6010:[diagnostics={os=FreeBSD, java.vendor=Oracle Corporation, java.version=1.8.0_202, java.vm.version=25.202-b08, lucene.version=8.0.0, os.arch=amd64, java.runtime.version=1.8.0_202-b08, source=flush, os.version=12.0-RELEASE-p3, timestamp=1557766748246}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]) Uninverting(_6z(8.0.0):C6622:[diagnostics={os=FreeBSD, java.vendor=Oracle Corporation, java.version=1.8.0_202, java.vm.version=25.202-b08, lucene.version=8.0.0, os.arch=amd64, java.runtime.version=1.8.0_202-b08, source=flush, os.version=12.0-RELEASE-p3, timestamp=1557766821254}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]) Uninverting(_71(8.0.0):C5449:[diagnostics={os=FreeBSD, java.vendor=Oracle Corporation, java.version=1.8.0_202, java.vm.version=25.202-b08, lucene.version=8.0.0, os.arch=amd64, java.runtime.version=1.8.0_202-b08, source=flush, os.version=12.0-RELEASE-p3, timestamp=1557766829833}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]) Uninverting(_72(8.0.0):C5439:[diagnostics={os=FreeBSD, java.vendor=Oracle Corporation, java.version=1.8.0_202, java.vm.version=25.202-b08, lucene.version=8.0.0, os.arch=amd64, java.runtime.version=1.8.0_202-b08, source=flush, os.version=12.0-RELEASE-p3, timestamp=1557766837529}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]) Uninverting(_73(8.0.0):C5187:[diagnostics={os=FreeBSD, java.vendor=Oracle Corporation, java.version=1.8.0_202, java.vm.version=25.202-b08, lucene.version=8.0.0, os.arch=amd64, java.runtime.version=1.8.0_202-b08, source=flush, os.version=12.0-RELEASE-p3, timestamp=1557766845144}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]) Uninverting(_74(8.0.0):C3048:[diagnostics={os=FreeBSD, java.vendor=Oracle Corporation, java.version=1.8.0_202, java.vm.version=25.202-b08, lucene.version=8.0.0, os.arch=amd64, java.runtime.version=1.8.0_202-b08, source=flush, os.version=12.0-RELEASE-p3, timestamp=1557766849549}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}])))}
INFO  (searcherExecutor-11-thread-1-processing-x:core0) [   x:core0] o.a.s.c.QuerySenderListener QuerySenderListener done.
INFO  (searcherExecutor-11-thread-1-processing-x:core0) [   x:core0] o.a.s.c.SolrCore [core0] Registered new searcher Searcher@8c5c9e1[core0] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_70(8.0.0):C60051:[diagnostics={os=FreeBSD, java.vendor=Oracle Corporation, java.version=1.8.0_202, java.vm.version=25.202-b08, lucene.version=8.0.0, mergeMaxNumSegments=-1, os.arch=amd64, java.runtime.version=1.8.0_202-b08, source=merge, mergeFactor=10, os.version=12.0-RELEASE-p3, timestamp=1557766812128}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]) Uninverting(_6o(8.0.0):C6151:[diagnostics={os=FreeBSD, java.vendor=Oracle Corporation, java.version=1.8.0_202, java.vm.version=25.202-b08, lucene.version=8.0.0, os.arch=amd64, java.runtime.version=1.8.0_202-b08, source=flush, os.version=12.0-RELEASE-p3, timestamp=1557766730928}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]) Uninverting(_6q(8.0.0):C6010:[diagnostics={os=FreeBSD, java.vendor=Oracle Corporation, java.version=1.8.0_202, java.vm.version=25.202-b08, lucene.version=8.0.0, os.arch=amd64, java.runtime.version=1.8.0_202-b08, source=flush, os.version=12.0-RELEASE-p3, timestamp=1557766748246}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]) Uninverting(_6z(8.0.0):C6622:[diagnostics={os=FreeBSD, java.vendor=Oracle Corporation, java.version=1.8.0_202, java.vm.version=25.202-b08, lucene.version=8.0.0, os.arch=amd64, java.runtime.version=1.8.0_202-b08, source=flush, os.version=12.0-RELEASE-p3, timestamp=1557766821254}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]) Uninverting(_71(8.0.0):C5449:[diagnostics={os=FreeBSD, java.vendor=Oracle Corporation, java.version=1.8.0_202, java.vm.version=25.202-b08, lucene.version=8.0.0, os.arch=amd64, java.runtime.version=1.8.0_202-b08, source=flush, os.version=12.0-RELEASE-p3, timestamp=1557766829833}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]) Uninverting(_72(8.0.0):C5439:[diagnostics={os=FreeBSD, java.vendor=Oracle Corporation, java.version=1.8.0_202, java.vm.version=25.202-b08, lucene.version=8.0.0, os.arch=amd64, java.runtime.version=1.8.0_202-b08, source=flush, os.version=12.0-RELEASE-p3, timestamp=1557766837529}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]) Uninverting(_73(8.0.0):C5187:[diagnostics={os=FreeBSD, java.vendor=Oracle Corporation, java.version=1.8.0_202, java.vm.version=25.202-b08, lucene.version=8.0.0, os.arch=amd64, java.runtime.version=1.8.0_202-b08, source=flush, os.version=12.0-RELEASE-p3, timestamp=1557766845144}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]) Uninverting(_74(8.0.0):C3048:[diagnostics={os=FreeBSD, java.vendor=Oracle Corporation, java.version=1.8.0_202, java.vm.version=25.202-b08, lucene.version=8.0.0, os.arch=amd64, java.runtime.version=1.8.0_202-b08, source=flush, os.version=12.0-RELEASE-p3, timestamp=1557766849549}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}])))}
INFO  (qtp349420578-16) [   x:core0] o.a.s.u.p.LogUpdateProcessorFactory [core0]  webapp=/solr path=/update params={commit=true}{add=[1, 3, 4, 5, 6, 7, 8, 12, 13, 17, ... (97957 adds)],commit=} 0 140797
INFO  (qtp349420578-20) [   x:core1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
INFO  (qtp349420578-20) [   x:core1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@42e13429 commitCommandVersion:0
INFO  (qtp349420578-20) [   x:core1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4f2f2289[core1] main]
INFO  (qtp349420578-20) [   x:core1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
INFO  (searcherExecutor-10-thread-1-processing-x:core1) [   x:core1] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@4f2f2289[core1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_1e(8.0.0):C6778:[diagnostics={os=FreeBSD, java.vendor=Oracle Corporation, java.version=1.8.0_202, java.vm.version=25.202-b08, lucene.version=8.0.0, os.arch=amd64, java.runtime.version=1.8.0_202-b08, source=flush, os.version=12.0-RELEASE-p3, timestamp=1557766860399}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]) Uninverting(_1f(8.0.0):C6493:[diagnostics={os=FreeBSD, java.vendor=Oracle Corporation, java.version=1.8.0_202, java.vm.version=25.202-b08, lucene.version=8.0.0, os.arch=amd64, java.runtime.version=1.8.0_202-b08, source=flush, os.version=12.0-RELEASE-p3, timestamp=1557766868779}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]) Uninverting(_1g(8.0.0):C5931:[diagnostics={os=FreeBSD, java.vendor=Oracle Corporation, java.version=1.8.0_202, java.vm.version=25.202-b08, lucene.version=8.0.0, os.arch=amd64, java.runtime.version=1.8.0_202-b08, source=flush, os.version=12.0-RELEASE-p3, timestamp=1557766877352}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]) Uninverting(_1h(8.0.0):C2786:[diagnostics={os=FreeBSD, java.vendor=Oracle Corporation, java.version=1.8.0_202, java.vm.version=25.202-b08, lucene.version=8.0.0, os.arch=amd64, java.runtime.version=1.8.0_202-b08, source=flush, os.version=12.0-RELEASE-p3, timestamp=1557766881149}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}])))}
INFO  (searcherExecutor-10-thread-1-processing-x:core1) [   x:core1] o.a.s.c.QuerySenderListener QuerySenderListener done.
INFO  (searcherExecutor-10-thread-1-processing-x:core1) [   x:core1] o.a.s.c.SolrCore [core1] Registered new searcher Searcher@4f2f2289[core1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_1e(8.0.0):C6778:[diagnostics={os=FreeBSD, java.vendor=Oracle Corporation, java.version=1.8.0_202, java.vm.version=25.202-b08, lucene.version=8.0.0, os.arch=amd64, java.runtime.version=1.8.0_202-b08, source=flush, os.version=12.0-RELEASE-p3, timestamp=1557766860399}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]) Uninverting(_1f(8.0.0):C6493:[diagnostics={os=FreeBSD, java.vendor=Oracle Corporation, java.version=1.8.0_202, java.vm.version=25.202-b08, lucene.version=8.0.0, os.arch=amd64, java.runtime.version=1.8.0_202-b08, source=flush, os.version=12.0-RELEASE-p3, timestamp=1557766868779}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]) Uninverting(_1g(8.0.0):C5931:[diagnostics={os=FreeBSD, java.vendor=Oracle Corporation, java.version=1.8.0_202, java.vm.version=25.202-b08, lucene.version=8.0.0, os.arch=amd64, java.runtime.version=1.8.0_202-b08, source=flush, os.version=12.0-RELEASE-p3, timestamp=1557766877352}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]) Uninverting(_1h(8.0.0):C2786:[diagnostics={os=FreeBSD, java.vendor=Oracle Corporation, java.version=1.8.0_202, java.vm.version=25.202-b08, lucene.version=8.0.0, os.arch=amd64, java.runtime.version=1.8.0_202-b08, source=flush, os.version=12.0-RELEASE-p3, timestamp=1557766881149}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}])))}
INFO  (qtp349420578-20) [   x:core1] o.a.s.u.p.LogUpdateProcessorFactory [core1]  webapp=/solr path=/update params={commit=true}{add=[378, 462, 616, 649, 819, 849, 857, 869, 880, 968, ... (21988 adds)],commit=} 0 29390

Are you running these nodes in cloud mode or local only mode? — MatsLindh
My intention is for it to be a local standalone instance as we had with Solr 3. — TomServo

TomServo TomServo · Accepted Answer · 2019-05-15T18:48:53

Doing the following fixed my problem(s):

Changed locktype to none (this is fine for my application)

Removed the write.lock file from both /data/index for each core.

Disabled UpdateLogging

Deleted the created /data/tlog folder and contents for each core.

I don't know exactly why the locktype stuff was giving me such a problem, but it would cause the index files to be deleted on startup in certain circumstances. Most likely time based.

The tlog/UpdateLogging was what was what was causing the other machines to run the delete and add queries on startup.

*This only worked once! We were able to copy all the index files to another machine and restart and it didn't delete the segments this time. But a few days later we got different behavior.

Now, it deletes all the segment files and then writes one segments_1 file. The log doesn't say why it's doing this. I'm not sure what to do, stuck again!

**Final solution was to use solr backup and restore API to load in a new index. Turns out you cannot edit or modify the files in the /data/index folder while solr is running, which was possible in our Solr 3 instance but not in Solr 8 apparently.

How to Prevent Solr from doing commit on restart that deletes the index

1 Answers