lucene indexing large database with multible tables

0
votes

I have 2 tables, out of this 1 table having huge data(15gb) of website text (not html) content. I want to index these 10 tables(having relation ship). eg: select a.id ,a.title,a.ipaddress, b.content from table1 a, table2 b where a.id=b.id;

here the problem is it is taking 70-80 hours to retrieve data from oracle database for indexing. I tried server side pagination like where rownum >1 and <20000 and increment each time. but still it is taking huge time becuase I am retriving clob data. So if I index two tables separate with separate C:\index1, table2 C:\index2. Is it possible to retrieve data using combination of two indexes.

which is is best? indexing separate or indexing as single? Note: my data in database will not update. I just need best way.

1
lucenelucene.net

1 Answers

1
votes

Is it possible to retrieve data using combination of two indexes.

Yes, it is possible to retrieve data from the two indexes.

Open an IndexReaders for each index. Call them reader1 and reader2. Then you can use a MultiReader to read the indices and append the content.

MultiReader multiReader = new MultiReader(reader1, reader2); IndexSearcher searcher = new IndexSearcher(multiReader);

Reference: How to perform search over independent index sets and merge results?

which is is best? indexing separate or indexing as single? Note: my data in database will not update. I just need best way.

Since the data in the DB will not change, I'd suggest that you use a single index.

On a side note, you might want to have a look at How to make indexing faster.