AWS ElasticSearch with Lambda and S3 doesn't add documents to index

Question

I have a bit of a mysterious issue: I have a lambda function which transports data from S3 bucket to AWS ES cluster. My lambda function runs correctly and reports the following:

All 6 log records added to ES

However added documents do not appear in AWS ElasticSearch index /_cat/indices?v

health status index uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   logs  3N2O9CqhSwCP6sj1QK5EQw   5   1          0            0      1.2kb          1.2kb

I'm using this lambda function https://github.com/aws-samples/amazon-elasticsearch-lambda-samples/blob/master/src/s3_lambda_es.js Lambda function's role has full permissions to ES cluster and S3 bucket. It can access S3 bucket because I can print out contents to Lambda's console log

Any ideas for further debugging are much appreciated!

Cheers

Are you sure you've configured you're esDomain correctly? i.e. are you sure you're checking the same cluster (in the same region) you're sending the documents to? — Val
It would also be nice to see what the content of the response body is when the request ends. There might be some useful info in there. Also why going through the trouble of reimplementing an ES client when you could simply require the official JS client and leverage it? — Val
Here we go. One interesting finding from the response body: type":"illegal_argument_exception","reason":"unknown setting [index.remote_addr] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"}] I suspect that lambda function was written for older version. Also request type is incorrect because for ES 6.3 it should be PUT instead of POST. Thank you! — thepolina
I would definitely use the official JS client as I stated above. — Val
Thanks for help! apparently my mapping and json structure from clf-parser library (in github example) didn't match — thepolina

Mark Bramnik Mark Bramnik · Accepted Answer · 2018-10-23T05:30:58

There can be many reasons for this. since you are asking about ideas for debugging, here are couple of them:

Add the console.log in postDocumentToES method of the lambda that shows where exactly does it connect
Try to extract the code from lambda and run it locally just to make sure it succeeds to send to elastic search (so that the code is correct at least)
Make sure that there are no "special restrictions" on index (like ttl for a couple of minutes or something), or, maybe something that doesn't allow inserting into the index.
How many ES servers do you have? Maybe there is a cluster of them and the replication is not configured correctly, so when you check the state of the index in one ES it doesn't actually have the documents but the other ES server could have these docs.

AWS ElasticSearch with Lambda and S3 doesn't add documents to index

1 Answers