Sunspot SOLR KeywordTokenizer - search won't return proper results

Question

So I'm using sunspot in my rails project, and I'm indexing song names. What I want to do is when I do a search for a specific term (for example "lose yourself"), in the top of my results list to appear the records that contain the exact match for the query (with spaces and everything).

To do that I decided to define a new fieldType in the schema.xml using KeywordTokenizerFactory like this (I was planning to use it alongside the normal text field with the StandardTokenizer, use both of these and boost the results that came out using the KeywordTokenized field):

<fieldType name="text_exact" class="solr.TextField" omitNorms="false">
  <analyzer>
    <tokenizer class="solr.KeywordTokenizerFactory"/>
    <filter class="solr.LowerCaseFilterFactory"/>
    <filter class="solr.TrimFilterFactory"/>
  </analyzer>
</fieldType>

and adding a dynamic field like this (not sure it should be multivalued for this purpose, maybe someone can enlighten me on this too):

<dynamicField name="*_exact" stored="false" type="text_exact" multiValued="true" indexed="true"/>

Now, in my song.rb file i have this in the searchable configuration:

text :song_name do
  self.name
end

text :song_name_exact, as: :song_name_exact do
  self.name
end

The problem is that when I try to search by using the song_name_exact field I get no results if my query contains spaces (so if i have a song called foo and I search for foo it will find it, but if i have one called foo bar and search for foo bar, the search query returns no results.

So first of all, I would ask if my approach to this is correct, and why isn't the search over the field with keyword tokenizer working properly?

What does your search query use like - you'll want a similar analyser to be used for that too — Frederick Cheung
If you're asking if I use the same analyzer for querying and indexing, the answer is yes (if you look at the fieldtype configuration you'll see that I haven't separated the query and the index analyzers, so they both use the same stuff) — Andrei S
What if you "wrap" the term in quotes before the search but using a regular text field? If you also need the non-exact matches you could do 2 searches... — brutuscat
I didn't try wrapping the term in quotes.. and also this seems like a hack.. I mean why bend sunspot/solr, when this is the way it would be supposed to work — Andrei S
Are you using standard query parser or the dismax query parser? — Siddhartha Reddy

Moustafa Samir Moustafa Samir · Accepted Answer · 2013-03-21T22:33:28

The reason why search for "lose yourself" only doesn't provide a result is that the token generated by KeywordTokenizerFactory is "lose youself" but the queryParse used by sunspot split your query into two tokens "lose" and "yourself"

Solution

so you should search for the whole phrase by adding double quotes to the search to be "\"lose yourself\""

Sunspot SOLR KeywordTokenizer - search won't return proper results

1 Answers