Duplicate SOLR Document Issue While Using Overwrite=True

Question

I am having an issue with temporary duplicate documents in my SOLR collection that are causing my user rankings system to be incorrect.

I am using SOLR version 4.8.1 so it is one of the latest builds. I am using XML to update the SOLR collection like described in this SOLR Documentation:

<add overwrite="true" commitWithin="#COMMIT_WITHIN.GLOBAL_VALUE#">
  <doc>
    <field name="END_USER_ID">#END_USER_ID#</field>
    <field name="TARGET_REGION_ID">#TARGET_REGION_ID#</field>
    <field name="POPULARITY_RANK">#POPULARITY_RANK#</field>
    <field name="VISIBILITY_SCORE">#VISIBILITY_SCORE#</field>
    <field name="POPULARITY_VISIBILITY_SCORES_ID">#POPULARITY_VISIBILITY_SCORES_ID#</field>
    <cfif #POP_VIS_SCORES_LAST_MODIFIED_DATETIME# NEQ "">
      <field name="POPULARITY_VISIBILITY_SCORES_DATE_MODIFIED">#POP_VIS_SCORES_LAST_MODIFIED_DATETIME#</field>
    </cfif>
  </doc>
</add>

As you can see from the code above, I am using the overwrite parameter (to have newer documents replace previously added documents with the same uniqueKey) in conjunction with the commitWithin parameter (to add the document within a certain time period). The uniqueKey in this case should be END_USER_ID and the time period should be 15 seconds; I have checked to make sure that the uniqueKey is defined in the appropriate schema.xml file and that multiValued is set to false for END_USER_ID.

So on my rankings page, there are several calls to our local SOLR server. For example:

http://localhost:8983/solr/pop_vis_scores/select/?q=TARGET_REGION_ID:50%20AND%20-POPULARITY_RANK:0&version=4.8&start=0&rows=1&indent=off&stats=true&stats.field=POPULARITY_RANK&sort=POPULARITY_RANK%20ASC&fl=[docid],END_USER_ID,POPULARITY_RANK&timeAllowed=8000

From my observations, when the commitWithin is set to 15000 milliseconds, the updated SOLR document is available right away but a duplicate SOLR document exists that reflects the older data. When the commitWithin is set to 500 milliseconds, it seems like the problem does not exist. Having said that, I would theorize the problem is still there but users cannot act quickly enough to see the duplicate documents. When I have thousands of users playing this game, I theorize that this problem may in fact still exist on a larger scale. In addition, it would be nice to set that commitWithin back to 15 seconds when the player base of the game increases.

Anybody face a similar issue before and if so, how would you go by solving it? Anybody have any recommendations? Thanks in advance!

Travis Walters Travis Walters · Accepted Answer · 2015-02-05T16:55:27

I assumed that when a SOLR document gets added to the collection within that given 15 second time window that the old document would get deleted at the same time as the new one would be inserted into the collection. It appears that this assumption was incorrect. I was able to exclude the user id from my queries to get more accurate statistical values when it came to rankings. For anybody experiencing a similar situation that I was in, I recommend not assuming that SOLR documents get deleted and updated at the same time.

Duplicate SOLR Document Issue While Using Overwrite=True

1 Answers