AWS SSE-KMS Encryption from Spark / Dataframes

1
votes

I have configured encryption enabled EMR cluster (properties in emrfs-site.xml) I am using dataframe savemode.append to write into S3n://my-bucket/path/ to save in s3.

But I am not able to see the object getting AWS KMS encrypted. However, when I do a simple insert from hive from EMR, I am able to see the objects getting aws kms encrypted.

How can I encrypt files from dataframe in S3 using sse kms?

1
apache-sparkencryptionamazon-s3emr

1 Answers

2
votes

The problem was we were using s3a to save the files from spark program to EMR. AWS officially doesn't support use of s3a on EMR. Though we were able to save data in S3, it was not encrypting the data. I tried using s3:// and s3n:// The encryption works with both.