How to update Solr documents on the Solr server side with custom handler / plugin

5
votes

I have a core with millions of records.
I want to add a custom handler which scan the existing documents and update one of the field based on a condition (age>12 for example).
I prefer doing it on the Solr server side for avoiding sending millions of documents to the client and back.
I was thinking of writing a solr plugin which will receive a query and update some fields on the query documents (like the delete by query handler).
I was wondering whether there are existing solutions or better alternatives.
I was searching the web for a while and couldn't find examples of Solr plugins which update documents (I don't need to extend the update handler).
I've written a plug-in which use the following code which works fine but isn't as fast as I need.
Currently I do: 

AddUpdateCommand addUpdateCommand = new AddUpdateCommand(solrQueryRequest); 
DocIterator iterator = docList.iterator(); 
SolrIndexSearcher indexReader = solrQueryRequest.getSearcher(); 
while (iterator.hasNext()) { 
   Document document = indexReader.doc(iterator.nextDoc()); 
   SolrInputDocument solrInputDocument = new SolrInputDocument(); 
   addUpdateCommand.clear(); 
   addUpdateCommand.solrDoc = solrInputDocument; 
   addUpdateCommand.solrDoc.setField("id", document.get("id")); 
   addUpdateCommand.solrDoc.setField("my_updated_field", new_value); 
   updateRequestProcessor.processAdd(addUpdateCommand); 
}

But this is very expensive since the update handler will fetch again the document which I already hold at hand.
Is there a safe way to update the lucene document and write it back while taking into account all the Solr related code such as caches, extra solr logic, etc?
I was thinking of converting it to a SolrInputDocument and then just add the document through Solr but I need first to convert all fields.
Thanks in advance, Avner

2
solrsolr4
You probably already know this, but in Solr, the act of updating a document really means replacing the original document with an updated document containing the changed value(s). The usual way to handle changes is to push from the source system into the index, usually based on a date or some other indication; not to somehow update the index locally. Can you give a bit more context for what you are trying to do? – icey502
I'm aware of the delete / add stuff. In my scenario there is data which exists only in the Solr index (calculated field). Later on I need to add some data to a field for all documents which answer a query based on the calculated field (in the example the "Age" field). Is it that complex to open a writer on the Solr side and change documents? – Avner Levy
I have not faced this exact scenario, but assuming the "trigger" for the changed value (and subsequent update) is itself an index update, maybe this link will be helpful: stackoverflow.com/questions/6593887/… – icey502
Thanks for the link, but my scenario is different. I need to change all documents in the Solr core and not only the one which is being updated. – Avner Levy

2 Answers

0
votes

I'm not sure whether the following is going to improve the performance, but thought it might help you.

Look at SolrEntityProcessor

Its description sounds very relevant to what you are searching for.

This EntityProcessor imports data from different Solr instances and cores. 
The data is retrieved based on a specified (filter) query. 
This EntityProcessor is useful in cases you want to copy your Solr index 
and slightly want to modify the data in the target index. 
In some cases Solr might be the only place were all data is available.

However, I couldn't find an out-of-the-box feature to embed your logic. So, you may have to extend the following class.

SolrEntityProcessor and the link to sourcecode

You may probably know, but a couple of other points.

1) Make the entire process exploit all the cpu cores available. Make it multi-threaded.

2) Use the latest version of Solr.

3) Experiment with two Solr apps on different machines with minimal network delay. This would be a tough call : 

same machine, two processes VS two machines, more cores, but network overhead.

4) Tweak Solr cache in a way that applies to your use-case and particular implementation.

5) A couple of more resources: Solr Performance Problems and SolrPerformanceFactors

Hope it helps. Let me know the stats despite this answer. I'm curious and your info might help somebody later.

0
votes

To point out where to put custom logic, I would suggest to have a look at the SolrEntityProcessor in conjunction with Solr's ScriptTransformer.

The ScriptTransformer allows to compute each entity after it is extracted from the source of a dataimport, manipulate it and add custom field values before the new entity is written to solr.

A sample data-config.xml could look like this

<?xml version="1.0" encoding="UTF-8" ?>
<dataConfig>

    <script>
    <![CDATA[
        function calculateValue(row)        {
            row.put("CALCULATED_FIELD", "The age is: " + row.get("age"));
            return row;
        }
    ]]>
    </script>

  <document>
    <entity name="sep" processor="SolrEntityProcessor" 
        url="http://localhost:8080/solr/your-core-name" 
        query="*:*"
        wt="javabin"
        transformer="script:calculateValue">
            <field column="ID" name="id" />
            <field column="AGE" name="age" />
            <field column="CALCULATED_FIELD" name="update_field" />
    </entity>
  </document>
</dataConfig>

As you can see, you may perform any data transformation you like and is expressible in javascript. So this would be a good point to express your logic and transformations.

You say one constraint maybe age > 12. I would handle this via the query attribute of the SolrEntityProcessor. You could write query=age:[* TO 12] so that only records with an age up to 12 would be read for the update.