I am indexing a large dataset 30 million rows and following each re-index (using a JDBC river) I am seeing inconsistencies in the total size of the index.
I am using: curl -XGET 'http://localhost:9200/index_name/_count'
and the results vary by as much as 100,000 results after each re-index.
I can't see any index errors in the log.
One possibility is that your refresh_interval setting is set to a high number. This option is used to reduce disk IO. Indexed results may only be available after this interval has expired.
You can also use the refresh API to force a refresh to take place. Like this:
curl -XPOST 'http://localhost:9200/index_name/_refresh'
See the elastic documentation for more details.
The algorithm used by ElasticSearch to count the number of entries in the index performs the operation in bounded memory usage. This leads to approximate results. To increase the precision you can set the
precision_threshold : AMOUNT_OF_ERROR
Still there is always a scope of 5% error in elastic search
