Indexing different fields with different fieldtype in Sunspot (Solr)

Question

I'd like to set up my indexing so that phonetic-matched results are given less weight than regular matches.

To do this, I've created two different fieldType sets in my schema.xml for text:

<fieldType name="text" class="solr.TextField" omitNorms="false">
  <analyzer>
    <tokenizer class="solr.StandardTokenizerFactory"/>
    <filter class="solr.StandardFilterFactory"/>
    <filter class="solr.LowerCaseFilterFactory"/>
    <filter class="solr.ISOLatin1AccentFilterFactory"/>
    <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/>
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="false"/>
  </analyzer>
</fieldType>
<fieldType name="text_phonetic" class="solr.TextField" omitNorms="false">
  <analyzer>
    <tokenizer class="solr.StandardTokenizerFactory"/>
    <filter class="solr.StandardFilterFactory"/>
    <filter class="solr.LowerCaseFilterFactory"/>
    <filter class="solr.ISOLatin1AccentFilterFactory"/>
    <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/>
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="false"/>
    <filter class="solr.PhoneticFilterFactory" encoder="DoubleMetaphone" inject="true"/>
  </analyzer>
</fieldType>

and created a dynamcicField that uses the phonetic factory:

<dynamicField name="*_phonetic" stored="false" type="text_phonetic" multiValued="true" indexed="true"/>

Now in my model I can do something like:

text :name, :as => :name_phonetic

and it works fine.

My question is, what is the best way to go about setting up a bunch of fields to use both the regular text field indexing alongside the phonetic one, with a higher boost to the first? I can just duplicate all of my indexing lines in my model, but is there a way for me to do this directly in the schema with the construct and have that available in the sunspot fulltext query?

Nick Zadrozny Nick Zadrozny · Accepted Answer · 2011-08-20T22:26:11

As you note, you can duplicate the lines in your searchable block for the fields you'd like indexed in multiple different ways. I'd actually recommend this, since you actually maintain some more granularity of fields (as will be shown below), and you have some nice Sunspot helpers like the inline :boost option.

That said, you can also make use of Solr's copyField directive in the schema. It looks something like this:

<copyField source="source_field" dest="dest_field" maxChars="N" />

The source field name may be a pattern, however your destination must be a single field. Furthermore, I believe the destination must be defined as its own field rather than be a name matched to a dynamicField.

Those constraints considered, you could set up something like this in your schema:

<fields>
  ...
  <field name="all_text_phonetic" stored="false" type="text_phonetic" multiValued="true" indexed="true"/>
  ...
</fields>

<copyField source="*_text" dest="all_text_phonetic" />
<copyField source="*_texts" dest="all_text_phonetic" />

To maintain granularity with your fields, you could set up a copyField directive for each incoming field. But then you'll have arguably more duplication than would be involved in creating separate lines in your searchable block.

So it's a tossup. But those are your options.

Indexing different fields with different fieldtype in Sunspot (Solr)

1 Answers