Elasticsearch Java using very high CPU and MEMORY

Question

Elasticsearch suddenly stopped working due high CPU usage, and now when I restart it, it keeps using around 100% CPU and 58% Memory it doesn't drop down. There is around 1.300.000 data linked to Elasticsearch. Using Linux server Ubuntu 15.04

default/elasticsearch

ES_HEAP_SIZE=2g (half of my memeory)
MAX_OPEN_FILES=65535
MAX_LOCKED_MEMORY=unlimited

limits.conf

elasticsearch - nofile 65535
elasticsearch - memlock unlimited

elasticsearch.yml

bootstrap.mlockall: true

curl http://localhost:9200/_nodes/hot_threads?pretty=true

::: {elasticnode}{wrdxzFcXT0aTXnp4R5KFbA}{127.0.0.1}{localhost/127.0.0.1:9300} Hot threads at 2016-03-01T10:24:09.884Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true: 93.7% (468.4ms out of 500ms) cpu usage by thread 'elasticsearch[elasticnode][clusterService#updateTask][T#1]' 10/10 snapshots sharing following 29 elements sun.nio.fs.UnixNativeDispatcher.readdir(Native Method) sun.nio.fs.UnixDirectoryStream$UnixDirectoryIterator.readNextEntry(UnixDirectoryStream.java :168) sun.nio.fs.UnixDirectoryStream$UnixDirectoryIterator.hasNext(UnixDirectoryStream.java:201) org.elasticsearch.index.translog.Translog$OnCloseRunnable.handle(Translog.java:726) org.elasticsearch.index.translog.Translog$OnCloseRunnable.handle(Translog.java:713) org.elasticsearch.index.translog.ChannelReference.closeInternal(ChannelReference.java:67) org.elasticsearch.common.util.concurrent.AbstractRefCounted.decRef(AbstractRefCounted.java: 64) org.elasticsearch.index.translog.TranslogReader.close(TranslogReader.java:143) org.apache.lucene.util.IOUtils.close(IOUtils.java:97) org.elasticsearch.index.translog.Translog.close(Translog.java:425) org.apache.lucene.util.IOUtils.close(IOUtils.java:97) org.apache.lucene.util.IOUtils.close(IOUtils.java:84) org.elasticsearch.index.engine.InternalEngine.closeNoLock(InternalEngine.java:990) org.elasticsearch.index.engine.Engine.close(Engine.java:1147) org.apache.lucene.util.IOUtils.close(IOUtils.java:97) org.apache.lucene.util.IOUtils.close(IOUtils.java:84) org.elasticsearch.index.shard.IndexShard.close(IndexShard.java:837) org.elasticsearch.index.IndexService.closeShardInjector(IndexService.java:443) org.elasticsearch.index.IndexService.removeShard(IndexService.java:416) org.elasticsearch.indices.cluster.IndicesClusterStateService.failAndRemoveShard(IndicesClus terStateService.java:743) org.elasticsearch.indices.cluster.IndicesClusterStateService.applyMappings(IndicesClusterSt ateService.java:364) org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterS tateService.java:164) org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalCluste rService.java:600) org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterServ ice.java:762) org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPriorit izedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231) org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPriorit izedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745)

0.1% (263.4micros out of 500ms) cpu usage by thread 'elasticsearch[elasticnode][transport_clie
nt_timer][T#1]{Hashed wheel timer #1}' 10/10 snapshots sharing following 5 elements java.lang.Thread.sleep(Native Method)

org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:445) org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:364) org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) java.lang.Thread.run(Thread.java:745)

Query example. This is the code in elasticsearch.php:

public function advancedquery($type, $query) {
    return $this->call($type . '/_search', 'POST', $query);
}

Any idea what is this about? I can provide any additional information as necessary.

Can you provide the code you're using to make the request? Although it sounds like it's redlining at 100% regardless of a request being made, is that correct? — iSkore
@iskore public function advancedquery($type, $query) { return $this -> call($type . '/_search', 'POST', $query); } This is the code in elasticsearch.php. Yes without making any requests usage is always very high PS. No we aren't using AWS ES. — Mert Alnuaimi
Hmm. Are you trying to run PHP in a Java environment or something? Not sure how this is being connected. Also, can you temporarily block the instance from making any requests to ElasticSearch? This will show if the instance is causing the spike. If the instance is not causing the spike, you might need to scale up your ElasticSearch cluster — iSkore
Not a problem. So lets get your architecture down first. You have an EC2 instance running Java - sending this request to ES? It looks like you actually have PHP running in a Java environment or something. Is this the case? — iSkore

iSkore iSkore · Accepted Answer · 2016-03-01T18:24:21

Based on the commentary. You have a cronjob running every 10 minutes that has node.js running a file in JavaScript. Based on your question, you have the JS file pinging a PHP file on the same server. On the same server you have a Java core running your posix stuff.

The answer to your CPU overload question is:

Choose one system. You have three environments running on the same instance. Java and PHP run on Apache, Node.JS on NGINX. And you shouldn't even cross Java and PHP. That's the problem. It's like running AutoCad, Final Cut Pro, and Unreal Game Engine all at the same time... doesn't work like that.

Condense you system into one language of your choice, supported by the AWS SDK's of course.

Elasticsearch Java using very high CPU and MEMORY

1 Answers