Read Files from S3 bucket to Spark Dataframe using Scala in Datastax Spark Submit giving AWS Error Message: Bad Request

0
votes

I'm trying to read CSV files which are on s3 bucket which is located in Mumbai Region.I'm trying to read the files using datastax dse spark-submit.

I tried changing hadoop-aws version to various other versions. Currently, hadoop-aws version is 2.7.3

spark.sparkContext.hadoopConfiguration.set("com.amazonaws.services.s3.enableV4", "true")

spark.sparkContext.hadoopConfiguration.set("fs.s3a.endpoint", "s3.ap-south-1.amazonaws.com")

spark.sparkContext.hadoopConfiguration.set("fs.s3a.access.key", accessKeyId)

spark.sparkContext.hadoopConfiguration.set("fs.s3a.secret.key", secretAccessKey)

spark.sparkContext.hadoopConfiguration.set("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")

val df = spark.read.csv("s3a://bucket_path/csv_name.csv")

Upon Executing, Following is the error which I'm getting,

Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 400, AWS Service: Amazon S3, AWS Request ID: 8C7D34A38E359FCE, AWS Error Code: null, AWS Error Message: Bad Request at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031) at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:297) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) at org.apache.spark.sql.execution.datasources.DataSource$.org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary(DataSource.scala:616) at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:350) at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:350) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) at scala.collection.immutable.List.foreach(List.scala:392) at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241) at scala.collection.immutable.List.flatMap(List.scala:355) at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:349) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178) at org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:533) at org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:412)

2
scalaapache-sparkamazon-s3apache-spark-sqldatastax-enterprise
check, your access key and secret access are valid or not – sumitya
Yes they are valid keys. This code works perfectly for US East Bucket – Chinmay R
You probably wan to run your s3 cleint as debug or you can search for a long time . Also beware you can either track your request using your request ID or even contact support to get what's wrong with that specific request ID especially if your have enabled server logging – leldo
Or check the resource you are accessing. Is it available in that region or not – sumitya

2 Answers

4
votes

Your signature V4 option is not applied. See This

Add the java option when you run the spark-submit or spark-shell.

spark.executor.extraJavaOptions=-Dcom.amazonaws.services.s3.enableV4=true
spark.driver.extraJavaOptions=-Dcom.amazonaws.services.s3.enableV4=true

Or, set the system property such as:

System.setProperty("com.amazonaws.services.s3.enableV4", "true");
1
votes

Thank you for all the help. I figured out from the answer of Lamanus that signature V4 option was not applied even after adding it in 

spark.sparkContext.hadoopConfiguration.set("com.amazonaws.services.s3.enableV4", "true")

So I added the following line and now the code works perfectly fine.

import com.amazonaws.SDKGlobalConfiguration

System.setProperty(SDKGlobalConfiguration.ENABLE_S3_SIGV4_SYSTEM_PROPERTY, "true")