Lucene and Multifield query

Question

I have an archive of university theses and publications indexed (with BM25 similarity) on Lucene (Java version). I have English document and Italian document, for this reason i have duplicate field like: pdf, pdf_en or like: titolo, titolo_en. When i have an italian document i fill italian field, otherwise i fill english filed.

Now i have a BooleanQuery with MultiFieldQueryParser, this is my code:

    String[] fieldsGEN={"url","autori","lingua","settore","pdfurl"};
    String[] fieldsITA={"titolo","tipologia","abstract","pdf"};

    String[] fieldsENG={"titolo_en","tipologia_en", "abstract_en","pdf_en"};

    MultiFieldQueryParser parserGEN = new MultiFieldQueryParser(version, fieldsGEN, analyzerIT);
    MultiFieldQueryParser parserITA = new MultiFieldQueryParser(version, fieldsITA, analyzerIT);
    MultiFieldQueryParser parserENG = new MultiFieldQueryParser(version, fieldsENG, analyzerENG);

    parserITA.setDefaultOperator(QueryParser.Operator.OR);
    parserITA.setDefaultOperator(QueryParser.Operator.OR);
    parserENG.setDefaultOperator(QueryParser.Operator.OR);

    Query query4 =parserGEN.parse(ricerca.ricerca);
    bq.add(query4, Occur.SHOULD);
    Query query2 =parserITA.parse(ricerca.ricerca);
    bq.add(query2, Occur.SHOULD);
    Query query3 =parserENG.parse(ricerca.ricerca);
    bq.add(query3, Occur.SHOULD);

If I search "anna" (Name of an author) the 3 query are:

Query: [titolo:anna tipologia:anna abstract:anna pdf:anna] 
Query: [titolo_en:anna tipologia_en:anna abstract_en:anna pdf_en:anna]
Query: [url:anna autori:anna lingua:anna settore:anna pdfurl:anna]

and I also authors without the name anna even if they are in the last position (about 3 document of 21 on 1000 indexed), I suppose that finds them in other fields.

Do you think the query is done well? the query can be improved? how? a search engine like google how it works on multifield search?

There is a better way to deal with multi-language field?

Thanks, Neptune.

I would probably combine them with a DisjunctionMaxQuery, since compounding scores for a query on hits in multiple languages doesn't make total sense to me. Other than that suggestion, I don't see any particular problem here right off. Is there some problem you want to deal with here? — femtoRgon
Why you suggest DisjunctionMaxQuery instead of BooleanQuery? what are the benefit? — Neptune
As per the documentation, it only scores as the maximum of subqueries, rather than compounding for hits on all language subqueries. One would assume only the best should be counted, and any others are likely false positives. Depends on your needs, but as a said, compounding scores across different languages seems unlikely to be beneficial. — femtoRgon

Mark Leighton Fisher Mark Leighton Fisher · Accepted Answer · 2014-01-06T23:50:08

Unless you have both translations for all documents, I would create 2 indexes -- 1 for each language, using the same field names for each index. You would then use a MultiReader with the search queries.

The problem with this approach is words that are spelled the same in each language but have different meanings between English and Italian. Apart from those words, I think that this architecture will be easier to understand as well as easier to interpret the results of.

Lucene and Multifield query

1 Answers