Unable to connect to AWS Elasticsearch from Spark

3
votes

I need to connect to AWS Elasticsearch service from my Spark app (structured streaming) using AWS access/secret keys. For example, S3 and Spark integration has a way to set access/secret keys in configuration https://docs.databricks.com/spark/latest/data-sources/aws/amazon-s3.html

I cannot find anything equivalent for Elasticsearch. I tried below code but it didn't work. 

 val writer = input.write
        .option("es.nodes",serverUrl)
        .option("es.net.https.auth.user", awsAccessKeyId)
        .option("es.net.https.auth.pass", awsSecretAccessKey)
        .option("es.nodes.wan.only", "true")
        .format("org.elasticsearch.spark.sql")
  writer.save("index/mapping")

Looks like "es.net.https.auth.xxx" are for basic authentication. I'm looking for AWS specific one. Any information is appreciated!

1
amazon-web-servicesapache-sparkelasticsearch
if you are okay and useful please care to vote-up. - Ram Ghadiyaram

1 Answers

1
votes

Along with your spark session awsAccessKeyId awsSecretAccessKey will be propogated if you have aws access to your spark cluster.

Test is, using spark shell try to read the bucket from s3 which has sample parquet file in it.

http auth user and password are the placeholders for es.net.https.auth.user es.net.https.auth.pass

for example if you are connecting to mysql (which is RDS is aws ) you need to pass jdbc user and password separately. similar terms your http authentication is also like that.

conclusion :

Out of my experience, there is no need to provide awsAccessKeyId awsSecretAccessKey separately since its implicit of you have access for that from your EMR cluster.

Note : if you want to or have to really set the access credentials you need to set that like below.... 

val hadoopConf = sc.hadoopConfiguration;
hadoopConf.set("fs.s3.impl", "org.apache.hadoop.fs.s3native.NativeS3FileSystem")
hadoopConf.set("fs.s3.awsAccessKeyId", myAccessKey)
hadoopConf.set("fs.s3.awsSecretAccessKey", mySecretKey)

not like 

.option("es.net.https.auth.user", awsAccessKeyId)