solr SolrIndexWriter was not closed prior to finalize(),​ indicates a bug -- POSSIBLE RESOURCE LEAK

3
votes

I have a Solr cloud (version 7.4) with 2 nodes (each one has 10Gb SSD hard and 256Gb memory and 50Gb heap) and 10 collections. Today I encountered this error about instagram_prime_archive collection : 

SolrIndexWriter was not closed prior to finalize(),​ indicates a bug -- POSSIBLE RESOURCE LEAK!!!

Error closing IndexWriter

null:org.apache.lucene.index.IndexNotFoundException: no segments* file found in LockValidatingDirectoryWrapper(NRTCachingDirectory(MMapDirectory@/home/solr/data/instagram_prime_archive_shard1_replica_n15/data/index.20190313041339667 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6b54ef37; maxCacheMB=48.0 maxMergeSizeMB=4.0)): files: [_14jqg.cfe, _14jqg.cfs, _14jqg.si, _14jqg_xcc.liv, _17wrq.cfe, _17wrq.cfs, _17wrq.si, _17wrq_8im.liv, _19qhp.cfe, _19qhp.cfs, _19qhp.si, _19qhp_3ik.liv, _19wns.cfe, _19wns.cfs, _19wns.si, _19wns_1yp.liv, _1a0lx.cfe, _1a0lx.cfs, _1a0lx.si, _1a0lx_9a.liv, _1a3u8.cfe, _1a3u8.cfs, _1a3u8.si, _1a3u8_7g.liv, _1a5fe.cfe, _1a5fe.cfs, _1a5fe.si, _1a5fe_es.liv, _1a7fu.cfe, _1a7fu.cfs, _1a7fu.si, _1a7fu_bg.liv, _1a9oo.cfe, _1a9oo.cfs, _1a9oo.si, _1a9oo_cv.liv, _1abp0.cfe, _1abp0.cfs, _1abp0.si, _1abp0_3fk.liv, _1afn9.cfe, _1afn9.cfs, _1afn9.si, _1afn9_98.liv, _1ajhu.cfe, _1ajhu.cfs, _1ajhu.si, _1ajhu_6r.liv, _1an7f.cfe, _1an7f.cfs, _1an7f.si, _1an7f_6t.liv, _1ap2m.cfe, _1ap2m.cfs, _1ap2m.si, _1ap2m_87.liv, _1au5v.cfe, _1au5v.cfs, _1au5v.si, _1au5v_72.liv, _1audy.cfe, _1audy.cfs, _1audy.si, _1audy_m.liv, _1aup4.cfe, _1aup4.cfs, _1aup4.si, _1aup4_m.liv, _1auza.cfe, _1auza.cfs, _1auza.si, _1auza_x.liv, _1avb8.cfe, _1avb8.cfs, _1avb8.si, _1avb8_u.liv, _1avgt.cfe, _1avgt.cfs, _1avgt.si, _1avgt_k.liv, _1avhd.cfe, _1avhd.cfs, _1avhd.si, _1avhd_8c.liv, _1avq1.cfe, _1avq1.cfs, _1avq1.si, _1avq1_13.liv, _1azqq.cfe, _1azqq.cfs, _1azqq.si, _1azqq_4.liv, _1azwu.cfe, _1azwu.cfs, _1azwu.si, _1azwu_1r.liv, _1b08s.cfe, _1b08s.cfs, _1b08s.si, _1b08s_5.liv, _1b0kq.cfe, _1b0kq.cfs, _1b0kq.si, _1b0kq_m.liv, _1b0ni.cfe, _1b0ni.cfs, _1b0ni.si, _1b0qa.cfe, _1b0qa.cfs, _1b0qa.si, _1b0tm.cfe, _1b0tm.cfs, _1b0tm.si, _1b0tm_1.liv, _1b0zh.cfe, _1b0zh.cfs, _1b0zh.si, _1b100.cfe, _1b100.cfs, _1b100.si, _1b10k.cfe, _1b10k.cfs, _1b10k.si, _1b11e.cfe, _1b11e.cfs, _1b11e.si, _1b11p.cfe, _1b11p.cfs, _1b11p.si, _1b11y.cfe, _1b11y.cfs, _1b11y.si, _1b128.cfe, _1b128.cfs, _1b128.si, _1b12s.cfe, _1b12s.cfs, _1b12s.si, _1b132.cfe, _1b132.cfs, _1b132.si, _1b13c.cfe, _1b13c.cfs, _1b13c.si, _1b13d.dii, _1b13d.dim, _1b13d.fdt, _1b13d.fdx, _1b13d.fnm, _1b13d.nvd, _1b13d.nvm, _1b13d.si, _1b13d_Lucene50_0.doc, _1b13d_Lucene50_0.pos, _1b13d_Lucene50_0.tim, _1b13d_Lucene50_0.tip, _1b13d_Lucene70_0.dvd, _1b13d_Lucene70_0.dvm, _1b13e.dii, _1b13e.dim, _1b13e.fdt, _1b13e.fdx, _1b13e.fnm, _1b13e.nvd, _1b13e.nvm, _1b13e.si, _1b13e_Lucene50_0.doc, _1b13e_Lucene50_0.pos, _1b13e_Lucene50_0.tim, _1b13e_Lucene50_0.tip, _1b13e_Lucene70_0.dvd, _1b13e_Lucene70_0.dvm, _1b13f.dii, _1b13f.dim, _1b13f.fdt, _1b13f.fdx, _1b13f.fnm, _1b13f.nvd, _1b13f.nvm, _1b13f.si, _1b13f_Lucene50_0.doc, _1b13f_Lucene50_0.pos, _1b13f_Lucene50_0.tim, _1b13f_Lucene50_0.tip, _1b13f_Lucene70_0.dvd, _1b13f_Lucene70_0.dvm, _1b14j.dii, _1b14j.dim, _1b14j.fdt, _1b14j.fdx, _1b14j.fnm, _1b14j.nvd, _1b14j.nvm, _1b14j.si, _1b14j_2.liv, _1b14j_Lucene50_0.doc, _1b14j_Lucene50_0.pos, _1b14j_Lucene50_0.tim, _1b14j_Lucene50_0.tip, _1b14j_Lucene70_0.dvd, _1b14j_Lucene70_0.dvm, _1b14k.dii, _1b14k.dim, _1b14k.fdt, _1b14k.fdx, _1b14k.fnm, _1b14k.nvd, _1b14k.nvm, _1b14k.si, _1b14k_2.liv, _1b14k_Lucene50_0.doc, _1b14k_Lucene50_0.pos, _1b14k_Lucene50_0.tim, _1b14k_Lucene50_0.tip, _1b14k_Lucene70_0.dvd, _1b14k_Lucene70_0.dvm, _1b14q.dii, _1b14q.dim, _1b14q.fdt, _1b14q.fdx, _1b14q.fnm, _1b14q.nvd, _1b14q.nvm, _1b14q.si, _1b14q_2.liv, _1b14q_Lucene50_0.doc, _1b14q_Lucene50_0.pos, _1b14q_Lucene50_0.tim, _1b14q_Lucene50_0.tip, _1b14q_Lucene70_0.dvd, _1b14q_Lucene70_0.dvm, _pqyd.cfe, _pqyd.cfs, _pqyd.si, _pqyd_3bhn.liv, _r9bx.cfe, _r9bx.cfs, _r9bx.si, _r9bx_33sj.liv, write.lock]
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:832)
at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:124)
at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:97)
at org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:257)
at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:131)
at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:344)
at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:284)
at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:234)
at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:950)
at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1168)
at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:633)
at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
at org.apache.solr.update.processor.SignatureUpdateProcessorFactory$SignatureUpdateProcessor.processAdd(SignatureUpdateProcessorFactory.java:193)
at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:261)
at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:188)
at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2539)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:709)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:515)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:531)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:760)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:678)
at java.lang.Thread.run(Thread.java:748)

I checked the htop and found that the number of threads was 9897 !!! enter image description here

There was also a lot of threads on the Thread Dump screen : enter image description here

What can be the reason for these Errors? And why has all this thread been created?

Could my code have a problem?

2
solrsolrcloud
Is your index stored on a local disk, or is it attached from a different host? The error seems to indicate that the index has disappeared from underneath Solr.. – MatsLindh
@MatsLindh All collections are stored on my 2 nodes (192.168.1.1 and 192.168.1.2) with 2 replicas – Alihossein shahabi

2 Answers

3
votes

Follow the steps to resolve issue:

Step 1: cd into your solr directory

Step 2: then go to your core data index directory

cd server/solr/<your-core-name>/data/index/

Step 3: then remove the lock file that is stopping the process

sudo rm write.lock

Step 4: cd into your solr directory again

Step 5: Now restart the solr instance by running following command

bin/solr restart

All Done! Now you can index again without problem

0
votes

It may be that SOLR is running using Java8 and the Oracle DB driver you are using does not support it (i.e. 'ojdbc10.jar'). Check the Oracle DB driver installed in the "server/lib" path; if SOLR is running with Java8 then you should have 'ojdbc8.jar' installed.