Storing words with apostrophe in Lucene index

5
votes

I've a company field in Lucene Index. One of the company names indexed is : Moody's

When user types in any of the following keywords,I want this company to come up in search results. 1.Moo 2.Mood 3.Moodys 4.Moody's

How should I store this index in Lucene and what type of Lucene Query should I use to get this behaviour?

Thanks.

2
lucenelucene.net
The answer depends on the way you want the rest of your index to be built: 1. Should pluralization be represented? i.e. are "Apple" and "Apples" distinct? 2. Do you want to keep apostrophes, or can they be obliterated? 3. Does a company name appear isolated, or inside a larger field? – Yuval F
thanks for ur comments...my response 1.No 2.I want Lucene not to keep apostrophes 3.Company name may appear isolted as well as in a larger field – Jimmy

2 Answers

9
votes

Based on your clarifications, I want to divide your question into two, and answer each in turn:

  1. How do I index words with apostrophes as equivalent to similar words without an apostrophe? e.g. mapping Moodys and Moody's to the same index term.
  2. How do I implement auto-complete search in Lucene - i.e. given an index, find documents using word prefixes, e.g. map Moo to Moodys ?

1 is relatively easy - Use a StandardToeknizer to create a token combining the apostrophe and s with the previous word, then a StandardFilter to remove the apostrophe and s. This will convert Moody's to Moody. A StandardAnalyzer does this and much more (lowercasing and stop word removal), which may be more than you need. Using a stemmer should take both Moodys and Moody to the same token. Try SnowBallFilter for this.

2 is harder: Lucene's PrefixQuery, to which Alan alluded, will only work when the company name is the first word in a field. You need something like the answer to this question about auto-complete in Lucene.

1
votes

The StandardAnalyser should work for 3 and 4, however won't work for 1 and 2.

Without writing your own (complex) text analyser, I would think about how you're expecting company names to be searched for. For example, basic lucene search syntax means that you could find "Moody's" if you search using wildcards: "Moo*" and "Mood*". Therefore, you might want to consider appending an "*" to the search term before submitting to lucene, however this might cause some confusion if the user isn't aware of this wildcard addition under the hood.