Index Writer in Append mode causing open deleted files in unix

Question

I recently upgraded our search application from Lucene 2.4 to Lucene 3.6.2 We run a job that pulls updates from database every 4 hours and updates the existing Lucene Index.

Code used for creating Index Writer -:

indexWriter = new IndexWriter(dirPath, new IndexWriterConfig(Version.LUCENE_36,
                    analyzer).setOpenMode(OpenMode.APPEND));

Post successfully adding documents to existing index we do following on Index Writer.

indexWriter.commit();
indexWriter.deleteUnusedFiles();
indexWriter.close();

Post above steps we do following on IndexSearcher

this.indexSrchr.getIndexReader().decRef();
this.indexSrchr.close();
// Create a new Index Searcher to point to updated index.
this.indexSrchr = new IndexSearcher(IndexReader.open(dir));

The Index Writer when opened in CREATE mode works perfectly fine but with APPEND mode it causes Open deleted files to be added every time i try the update on index.

Unix Command Used -:

lsof -u user | grep delete.

Can someone please help me if there is an issue at lucene level itself or we are doing something wrong?

I was able to solve the issue as somewhere in the code we were not explicitly doing a decrease reference on the indexReader

this.indexSrchr.getIndexReader().decRef()

But can someone help me understand why

this.indexSrchr.close()

does not close the underlying index reader of the Index Searcher ?

Mihai Soloi Mihai Soloi · Accepted Answer · 2013-06-25T15:32:12

Using CREATE you create a new index all the time and that's why you don't experience the issue. When you are APPENDing to the index it recognizes the previous one of the files already indexed as deleted.

If you don't close the IndexReader after you read through the dir the files are still being accessed so they can't be deleted by deleteUnusedFiles.

Index Writer in Append mode causing open deleted files in unix

1 Answers