approach for creating lucene indexes

Question

I am implementing search feature for a news website. On that website ,users submit news articles containing title and text, currently these articles are inserted directly into a database.I heard that full text searching inside a database containing long..long text would not be efficient.

so i tried using lucene for indexing and searching. i am able to index full database with it and also able to search the content.But i am not sure if i am using the best approach.

Here is my indexer class :

public class LuceneIndexer {
    public static void indexNews(Paste p ,IndexWriter indexWriter) throws IOException {

        Document doc = new Document();
        doc.add(new Field("id", p.getNewsId(), Field.Store.YES, Field.Index.NO));
        doc.add(new Field("title", p.getTitle(), Field.Store.YES, Field.Index.TOKENIZED));
        doc.add(new Field("text", p.getNewsRawText(), Field.Store.YES, Field.Index.UN_TOKENIZED));
        String fullSearchableText = p.getTitle() + " " + p.getNewsRawText();

        doc.add(new Field("content", fullSearchableText, Field.Store.NO, Field.Index.TOKENIZED));
        indexWriter.addDocument(doc);
    }

    public static void rebuildIndexes() {


        try {
            System.out.println("started indexing");
            IndexWriter w = getIndexWriter();
            ArrayList<News> n = new GetNewsInfo().getLastPosts(0);
            for (News news : n) {
                indexNews(news,w );
            }
             closeIndexWriter(w);
             System.out.println("indexing done");
        } catch (Exception e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }


    }

    public static IndexWriter getIndexWriter() throws CorruptIndexException, LockObtainFailedException, IOException {

        IndexWriter indexWriter = new IndexWriter(GlobalData.LUCENE_INDEX_STOREAGE, new StandardAnalyzer(), true);
        return indexWriter;
    }

    public static void closeIndexWriter(IndexWriter w) throws CorruptIndexException, IOException {
        w.close();

    }

Is above code efficient ?

I think i should add a document into index when it is submitted by user instead of indexing full database again.

Do i need to create new IndexWriter every time an article is submitted?
is that efficient to open and close an IndexWriter very frequently?

Rhand Rhand · Accepted Answer · 2012-07-27T11:54:22

You are right that you don't need to readd every document to the index, you only need to add new ones, the rest will remain in the index.

But then you do need to create a new IndexWriter every time. If you prefer you can use a service or something which keeps an IndexWriter alive, but the opening and closing does not take much time. If you do reuse an IndexWriter make sure that you use indexWriter.commit() after each adding.

approach for creating lucene indexes

2 Answers