The encryption method specified is not supported error when writing to s3 from spark job using Hadoop s3a

0
votes

I am trying to write to s3 bucket which is KMS Server-side encrypted with a spark job running on an emr cluster.

I am passing these configs with spark-submit command:

"spark.hadoop.fs.s3a.fast.upload=true",
"spark.hadoop.fs.s3a.access.key=XXXXXXXX",
"spark.hadoop.fs.s3a.secret.key=XXXXXXXX",
"spark.hadoop.fs.s3a.server-side-encryption-algorithm=SSE-KMS",
"spark.hadoop.fs.s3a.fast.upload=true",
"spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem"

But I am getting this error:

com.amazonaws.services.s3.model.AmazonS3Exception: The encryption method specified is not supported

Writes are working fine for AES256 encryption but with KMS it is creating the issue.

  • EMR Version: emr-5.30.0,
  • Spark Version: 2.4.0,
  • Hadoop Version on EMR: 2.8.5,
  • Aws-Java-Sdk: 1.11.835

With this link, I got to know KMS Was not supported in Hadoop till Hadoop 3.0.0: Doesn't Spark/Hadoop support SSE-KMS encryption on AWS S3

I tried passing external jars of org.apache.hadoop:hadoop-aws:3.0.0 with spark-submit command but still same error came up.

What am I missing here or doing wrong?

1
amazon-web-servicesapache-sparkhadoopamazon-s3encryption

1 Answers

0
votes

@Aayush Is it possible that you haven't specified the KMS key with fs.s3a.server-side-encryption.key ?

I am using s3 encryption with KMS keys just fine on hadoop 2.9.1 so it should work on 3.0 and likely also work on 2.8.5.