Elasticsearch Indexing Speed Degrade with the Time

0
votes

I am doing some performance tuning in elastic search for my project and I need some help in improving the elastic search indexing speed. I am using ES 5.1.1 and I have 2 nodes setup with 8 shards for the index. I have the servers for 2 nodes with 16GB RAM and 12CPUs allocated for each server with 2.2GHz clock speed. I need to index around 25,000,000 documents within 1.5 hours, which I am currently doing in around 4 hours. I have done the following config changes to improve the indexing time.

  • Setting ‘indices.store.throttle.type’ to ‘none’
  • Setting ‘refresh_interval’ to ‘-1’
  • Increasing ‘translog.flush_threshold_size’ to 1GB
  • Setting ‘number_of_replicas’ to ‘0’
  • Using 8 shards for the index
  • Setting VM Options -Xms8g -Xmx8g (Half of the RAM size)

I am using the bulk processor to generate the documents in my java application and I’m using the following configurations to setup the bulk processor.

Bulk Actions Count : 10000
Bulk Size in MB : 100
Concurrent Requests : 100
Flush Interval : 30

Initially I can index around 356167 in the first minute. But with the time, It decreases and after around 1 hour its around 121280 docs per minute.

How can I keep the indexing rate steady over the time? Is there any other ways to improve the performance?

1
elasticsearch

1 Answers

0
votes

I highly encourage not to change configuration parameters like the translog flush size, the throttling, unless you know what you are doing (and this does not mean reading some blog post on the internet :-)

Try a single shard per server and especially reduce the bulk size to something like 10MB. 100MB * 100 concurrent requests means you need 10GB of heap to deal with those (without doing anything else). I suppose not all of the documents get indexed because of your rejected tasks in your threadpools.

Start small and get bigger instead of starting big but not have any insight in your indexing.