I have a text box in a search form that I want to attach a combobox / autocomplete widget to. As users type, I want to auto-suggest relevant place names. So if a user types "Ca", suggest Cambodia, Cameroon, Canada, Cape Verde, etc, ranked higher than North Carolina and South Carolina. If a user types "Sea", return items such as Red Sea, Black Sea, etc, but perhaps not Chelsea (if at all, this should be scored lower). Our database of place names is very granular and complex, with a lot of data and a lot of alternate names / translations for places. The data is stored in SQL Server and we use Entity Framework as the data access layer. Needless to say, effectively querying our Places entity aggregate using LINQ to Entities is slow and inefficient.
Rather than carving custom SQL and indexes to optimize database searches, I am looking at Lucene.Net. Today is my first day testing it out. Most of the Lucene help I've read use a StandardAnalyzer to index. I was having some trouble using that for a couple of my tests. For example, consider the following:
var searchTerms = new[] { "Ca", "China", "Sea", };
searchTerms = searchTerms.Concat(searchTerms.Select(x => x.ToLower())).ToArray();
var reader = IndexReader.Open(_directory, true);
foreach (var searchTerm in searchTerms)
{
var searcher = new IndexSearcher(reader);
var query1 = new WildcardQuery(new Term("OfficialName", string.Format("*{0}*", searchTerm)));
var query2 = new TermQuery(new Term("OfficialName", searchTerm));
var query3 = new QueryParser(Version.LUCENE_30, "OfficialName", new SimpleAnalyzer()).Parse(searchTerm);
var query4 = new PrefixQuery(new Term("OfficalName", searchTerm));
var query5 = new BooleanQuery();
query5.Add(query1, Occur.SHOULD);
query5.Add(query2, Occur.SHOULD);
query5.Add(query3, Occur.SHOULD);
query5.Add(query4, Occur.SHOULD);
var queryToRun = query5;
var results = searcher.Search(queryToRun, int.MaxValue);
var hits = results.ScoreDocs;
The above code just tries out normal- and lower-cased versions of terms. Interestingly the "Ca" query returns no results, but "ca" returns a ton of them -- Africa, North America, etc. I think I read somewhere that the standard analyzer differentiates terms based on case, so this may be why..? The other search terms return what one might expect.
When the same data is indexed using a keyword analyzer, the results are quite different. One weird thing is that "china" only returns 1 result, "Uchinada-machi". I would have expected it to also return "China" and "East China Sea". Also "sea" returns results like "Royal Borough of Kensington and Chelsea" and "Swansea City and County", but none of the other expected results.
So how should I go about this? Should I have different indexes of the text for different analyzers? Do I need to query against a document field with lowercased text? I read about using NGram tokenizers, but they no longer seem to be in the Lucene.Net.Analysis namespace.
