Why does my solr slave index keep growing?

Question

I have a 5-core solr 1.4 master that is replicated to another 5-core solr using solr replication as described here. All writes are done against the master and replicated to the slave intermittently. This is done using the following sequence:

Commit on each master core
Replicate on each slave core
Optimize on each slave core
Commit on each slave core

The problem I am having is that the slave seems to be keeping around old index files and taking up ever more disk space. For example, after 3 replications, the master core data directory looks like this:

$ du -sh *
145M    index

But the data directory on the slave of the same core looks like this:

$ du -sh *
300M    index
144M    index.20100621042048
145M    index.20100629035801
4.0K    index.properties
4.0K    replication.properties

Here's the contents of index.properties:

#index properties
#Tue Jun 29 15:58:13 CDT 2010
index=index.20100629035801

And replication.properties:

#Replication details
#Tue Jun 29 15:58:13 CDT 2010
replicationFailedAtList=1277155032914
previousCycleTimeInSeconds=12
timesFailed=1
indexReplicatedAtList=1277845093709,1277155253911,1277155032914
indexReplicatedAt=1277845093709
replicationFailedAt=1277155032914
lastCycleBytesDownloaded=150616512
timesIndexReplicated=3

The solrconfig.xml for this slave contains the default deletion policy:

[...]
<mainIndex>
    <unlockOnStartup>false</unlockOnStartup>
    <reopenReaders>true</reopenReaders>
    <deletionPolicy class="solr.SolrDeletionPolicy">
        <str name="maxCommitsToKeep">1</str>
        <str name="maxOptimizedCommitsToKeep">0</str>
    </deletionPolicy>
</mainIndex>
[...]

What am I missing?

To closers: I really don't think this belongs on serverfault.com — Mauricio Scheffer

Pascal Dimassimo Pascal Dimassimo · Accepted Answer · 2010-06-30T13:40:28

It is useless to commit and optimize on the slaves. Since all the write operations are done on the master, it is the only place where those operations should occur.

This may be the cause of the problem: since you do an additional commit and optimize on the slaves, it keeps more commit points on the slaves. But this is only a guess, it should be easier to understand what happens with your full solrconfig.xml on both the master and the slaves.

Why does my solr slave index keep growing?

3 Answers