AWS elasticsearch lost documents

Question

I have an AWS Elasticsearch instance with only one node. After inserting some records and getting CONFIRMATION back from the AWS-ES, the documents get lost. I know that there is a delay because of shards refresh, but this is not the case because we check two or three days after inserting the record. In case it makes a difference, I'm using Jest as the HTTP client.

After inserting the record, I get a response like {"_index":"XXXXXXXXX","_type":"document","_id":"354d7537-3062-4d0c-ba31-26a7ba686919","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"created":true}

When I try to search again using the _id, it is not there. {"_index":"XXXXXXXXXX","_type":"document","_id":"354d7537-3062-4d0c-ba31-26a7ba686919","found":false}

Answer 1

After some research, I came to the conclusion that an interruption happened on the AWS-Elasticsearch side, restart, power failure or some other failure that led to memory being wiped. Records inserted to Elasticsearch are kept in memory for some time before they are persisted permanently to disk. If an interruption happens to the elastic search during this period, records that are in the memory get lost. The surprising part here is how AWS-Elasticsearch was interrupted. From our records, we haven't done anything to the instance during this period of time. Also, the lost records are all inserted after each other, which strengthens the interruption assumption.