ElasticSearch indices become empty randomly

Question

ElasticSearch indices become empty randomly, but most of times, it occurs after deploying the application built with Rails.

Here some info about ElastiSearch:

curl -XGET 'http://localhost:9200/_nodes?pretty'

curl -XGET 'http://localhost:9200/_cluster/health?pretty'

curl 'localhost:9200/_cat/indices?v'

Recurring stack trace in /var/log/elasticsearch/elasticsearch.log:

[2014-11-21 16:46:05,204][WARN ][cluster.service          ] [Joe Cartelli] failed to reconnect to node [Joe Cartelli][AvwiuMNRRUywfTyii7gI0A][ip-172-31-19-137][inet[/172.31.19.137:9300]]
org.elasticsearch.transport.ConnectTransportException: [Joe Cartelli][inet[/172.31.19.137:9300]] connect_timeout[30s]
    at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:727)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:656)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:624)
    at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:146)
    at org.elasticsearch.cluster.service.InternalClusterService$ReconnectToNodes.run(InternalClusterService.java:518)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.net.ConnectException: Connection refused: /172.31.19.137:9300
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
    at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
    at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
    at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
    at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
    at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    ... 3 more

Obs: The 172.31.19.137 IP address is the private IP of EC2 instance on AWS.

EDIT:

The erros stills occurs and I had new information about the issue.

After 'data gone', I looked at elasticsearch.log and Stack Trace was printed many times. I don't know if this error happens before or after the 'data gone'. Note that the index name is products_production_20141130122307375.

Running the command to list all Elasticsearch indexes curl 'localhost:9200/_cat/indices?v', printed the products index but with different number: products_production_20141XXXXXXXXXX (I didn't wrote down the exact number).

So, why the name of indexes weren't same?

It's possible, for some strange reason for me, Rails/Searchkick recreate the index and lose the data?

EDIT 2

The problem occurs after ElasticSearch log this:

[2014-12-02 00:47:10,698][INFO ][cluster.metadata         ] [Tyr] [products_production_20141202004040313] update_mapping [product] (dynamic)
[2014-12-02 00:58:02,173][INFO ][cluster.metadata         ] [Tyr] [products_production_20141202005801257] creating index, cause [api], shards [5]/[1], mappings [_default_]
[2014-12-02 00:58:46,064][INFO ][cluster.metadata         ] [Tyr] [products_production_20141202005801257] update_mapping [product] (dynamic)
[2014-12-02 00:58:48,371][INFO ][cluster.metadata         ] [Tyr] [products_production_20141202004040313] deleting index

It looks like that searchkick recreates the index and loses the products of older index.

Where is the data stored? It sounds like your EC2 instance goes away, and the data is wiped there. Separately, make sure your Rails application does not wipe out the indices (unrelated to the connectivity, which suggests the former issue). — pickypg
I don't know what's with the downvotes, but I also don't know why it's happening. Since it happens with the Rails application deployment, I suspect people want to see more details related to it. I'm still assuming that it wipes the data for some reason. — pickypg
anyway @Rodrigo can elaborate more over the question, but that isn't a reason for a downvote! — pedrommuller

dferrazm dferrazm · Accepted Answer · 2014-12-10T19:42:11

Maybe this issue could be related to your problem:

https://github.com/ankane/searchkick/issues/307

ElasticSearch indices become empty randomly

1 Answers