Lucene can't find documents after update

7
votes

It seems that whenever I update an existing document in the index (same behavior for delete / add), it can't be found with a TermQuery. Here's a short snippet:

iw = new IndexWriter(directory, config);

Document doc = new Document();
doc.add(new StringField("string", "a", Store.YES));
doc.add(new IntField("int", 1, Store.YES));

iw.addDocument(doc);

Query query = new TermQuery(new Term("string","a"));

Document[] hits = search(query);
doc = hits[0];
print(doc);

doc.removeField("int");
doc.add(new IntField("int", 2, Store.YES));

iw.updateDocument(new Term("string","a"), doc);

hits = search(query);
System.out.println(hits.length);
System.out.println("_________________");

for(Document hit : search(new MatchAllDocsQuery())){
    print(hit);
}

This produces the following console output:

stored,indexed,tokenized,omitNorms,indexOptions=DOCS_ONLY<string:a>
stored<int:1>
________________
0
_________________
stored,indexed,tokenized,omitNorms,indexOptions=DOCS_ONLY<string:a>
stored<int:2>
________________

It seems that after the update, the document (rather the new document) in the index and gets returned by the MatchAllDocsQuery, but can't be found by a TermQuery.

Full sample code available at http://pastebin.com/sP2Vav9v

Also, this only happens (second search not working) when the StringField value contains special characters (e.g. file:/F:/).

2
javalucene
Are you not missing iw.commit() ? - mindas
Doesn't change anything. Tried it aleady. Also, the search opens a new reader from the writer each time: DirectoryReader reader = DirectoryReader.open(iw, true); - Michael
I seem to be hitting a similar problem. Which version of Lucene is this? - carlspring
Tried it with a number of versions between 4_6 and 4_10. - Michael

2 Answers

4
votes

The code which you have referenced in pastebin doesn't find anything because your StringField is nothing but a stopword (a). Replacing a with something which is not a stopword (e.g. ax) makes both searches to return 1 doc.

You would also achieve the correct result if you were to construct StandardAnalyzer with empty stopword set (CharArraySet.EMPTY_SET) yet still using a for StringField. This wouldn't work for file:/F:/ though.

However, the best solution is this case would be to replace StandardAnalyzer with KeywordAnalyzer.

1
votes

I could get rid of this by recreating my working directory after all indexing operations : create a new directory just for this indexing operations named "path_dir" for example. If you have updated then call the following operations and do all of your previous works again.

StandardAnalyzer analyzer = new StandardAnalyzer(Version.LUCENE_46);
FSDirectory dir;
try {
    // delete indexing files :
    dir = FSDirectory.open(new File(path_dir));
    IndexWriterConfig config = new IndexWriterConfig(Version.LUCENE_46, analyzer);
    IndexWriter writer = new IndexWriter(dir, config);
    writer.deleteAll();
    writer.close();
    } catch (IOException e) {
        e.printStackTrace();
    }

However, note that this way will be very slow if you are handling big data.