Apache Solr Query Building

Question

I am a newbie to Apache Solr. I am trying to figure out the tokenizer, filter, and query parameters for the following query, but haven't been able to figure out if it's possible yet (still reading through all the documentation):

I have two fields - title and description. We want to do a search where:
1. Matches from title have more relevance than from description.
2. Complete word matches take precedence over all others (for query kit, kit takes precedence over kitchen).
3. An index entry that begins with the query field takes preference over one that just contains the field (for query goo, good takes precedence over Magoo).

Is this even possible? If so, how do I do this?

MatsLindh MatsLindh · Accepted Answer · 2018-08-24T18:46:38

Weighting between fields isn't an issue that tokenizers or filters are concerned with - their job is to take some input text, split it into tokens (tokenizers) and then run it through a sequence of processing steps (filters).

The edismax and dismax query parsers have a parameter named qf that allows you to give a list of fields that should be queried, and give a separate weight for each one - allowing you to tune exactly how much weight to give to each field. qf=title^5 description would weigh a hit in the field title five times higher than a field in description - all things else being identical (but they usually aren't identical, since you're not indexing the same content into both fields).

And that's the reason why scoring isn't an exact science, so if you want to have some sort of relevancy score used (i.e. different words hit will give different scores), you'll have to tweak these weights to fit the rank you're looking for. Appending debugQuery=true to a query is very helpful when you're tweaking scoring, since it'll show you exactly how much each term is contributing to the end score for a document.

Your first criteria, title vs description is solved by having a TextField with a StandardTokenizer and a lowercasing filter (and depending on what you're looking for, optionally stemming, synonyms, etc.).

You'll also (probably) want a lowercasefilter in the examples given below, but I've omitted it to keep the examples compact.

Your second case is solved by having a second field type that has an EdgeNGramFilter, and then having two new fields - title_edge and description_edge that uses this field type.

Both this and the NGramFilter example below uses the type="index" attribute, since it usually only makes sense to expand ngrams when indexing. Otherwise any two words starting with (or for NGram Filter, containing) identical letters would give a match.

<analyzer type="index">
  <tokenizer class="solr.StandardTokenizerFactory"/>
  <filter class="solr.EdgeNGramFilterFactory" minGramSize="2" maxGramSize="40" />
</analyzer>
<analyzer type="query">
  <tokenizer class="solr.StandardTokenizerFactory"/>
</analyzer>

The third criteria is solved by having a third set of fields, title_ngram and description_ngram that has an NGramFilter in its sequence:

<analyzer type="index">
  <tokenizer class="solr.StandardTokenizerFactory"/>
  <filter class="solr.NGramFilterFactory"/>
</analyzer>
<analyzer type="query">
  <tokenizer class="solr.StandardTokenizerFactory"/>
</analyzer>

Be aware that an NGramFilter will result in a lot of tokens being generated, require more storage and making searches process more tokens when generating a match. This may or may not be relevant for your use case.

That being said, there's something to be said about matching inner terms in words - especially very short strings. They might give results where the user isn't able to understand why the document was matched, as it might be a small match (a single letter while typing a query) somewhere. Someone searching for just "c" to find something about the programming language, will get every hit that has a word that contains c (but if you've boosted your fields properly, the exact hit should be at the top, luckily).

Apache Solr Query Building

1 Answers