Solr Split string and storing into multiple fields while Indexing

2
votes

Right now I have a field that is getting indexed in the below format,

“my_field”:”Abc&Deo&Efg”

There can be "n" number of items separated by "&", is there any way to split this field with & and store in separate field while Indexing in solr, field name can be the value itself in solr.

2
solr

2 Answers

3
votes

You will have to do this in the update chain. Use a ScriptUpdateProcessor then write some Javascript to do this

<processor class="solr.StatelessScriptUpdateProcessorFactory">
    <str name="script">updateProcessor.js</str>
</processor>

In update Processor script (in your conf directory):

    function processAdd(cmd) {
        doc = cmd.solrDoc;  // org.apache.solr.common.SolrInputDocument
        field= doc.getFieldValue("my_field");

        // tokenize your string here on the & separate then put tokens into new field, which could be a multivalue
        doc.setField("mySplitField", token);
    }

The question is why you want to do this when you can simply tokenize on '&' when indexing then each component is searchable.

Here is some more info: https://dutchweballiance.nl/techblog/introducing-the-solr-scriptupdateprocessor/

1
votes

Yes, you could do that with Regular Expression Pattern Tokenizer

I did a quick test by adding to schema.xml

    <field name="my_field" type="my_field_type" indexed="true" stored="true" required="true" multiValued="false" /> 
    <fieldType name="my_field_type" class="solr.TextField" positionIncrementGap="100">
      <analyzer>
        <tokenizer class="solr.PatternTokenizerFactory" pattern="&amp;"/>
      </analyzer>
    </fieldType>

So, basically the trick could be done with tokenizer, that will split data by some needed symbol, in your case it's ampersand.

enter image description here