aws s3 | bucket key enabled

Question

S3 has recently announced "bucket_key_enabled" option to cache the kms key used to encrypt the bucket contents so that the number of calls to the kms server is reduced.

https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-key.html

So if that the bucket is configured with

server side encryption enabled by default
use a kms key "key/arn1" for the above
by selecting "enable bucket key", we are caching "key/arn1" so that each object in this bucket does not require a call to kms server (perhaps internally it has time-to-live etc but the crust is that, this key is cached and thus kms limit errors can be avoided)

Given all that, what is the point of overriding kms key at object level and still having this "bucket_key_enabled" set?

Eg :

bucket/       -> kms1 & bucket_key_enabled
bucket/prefix1 -> kms2 & bucket_key_enabled

Does s3 actually cache the object-key to kms-key map?

To give you the context, I currently have the application which publishes data to the following structure

bucket/user1 
bucket/user2

While publishing to these buckets, it explicitly passed kms key assigned per user for each object upload.

bucket/user1/obj1 with kms-user-1
bucket/user1/obj2 with kms-user-1
bucket/user1/obj3 with kms-user-1


bucket/user2/obj1 with kms-user-2
bucket/user2/obj2 with kms-user-2
bucket/user2/obj3 with kms-user-2

if s3 is smart enough to reduce this to the following map,

bucket/user1 - kms-user-1
bucket/user2 - kms-user-2

All I have to do is, upgrade the sdk library to latest version and add a withBucketKeyEnabled(true) to the putObjectRequest in the s3Client wrapper we have.

Let me know how it works internally so that we can make use of this feature wisely.

Vinodhini Chockalingam Vinodhini Chockalingam · Accepted Answer · 2020-12-17T13:07:59

I finally went with upgrading the sdk to latest version and passing withBucketKeyEnabled(true) to putObject API calls.

I was able to prove with cloud trail that the number of calls to kms server is the same regardless of encryption and bucketKeyEnabled set at bucket level or at "each" object level.

kms-key and bucketKeyEnabled=true at bucket level. No encryption option is passed at putObject() call

Calls made to GenerateDataKey() = 10

Calls made to Decrypt() = 60

No encryption settings at s3 bucket. For each putObject() call, I am passing kms-key and bucketKeyEnabled=true.

PutObjectRequest(bucketName, key, inputStream, objectMetadata)
.withSSEAwsKeyManagementParams(SSEAwsKeyManagementParams(keyArn))
.withBucketKeyEnabled<PutObjectRequest>(true)

Calls made to GenerateDataKey() = 10

Calls made to Decrypt() = 60

With this option disabled like below,

PutObjectRequest(bucketName, key, inputStream, objectMetadata)
.withSSEAwsKeyManagementParams(SSEAwsKeyManagementParams(keyArn))

Calls made to GenerateDataKey() = 10011

Calls made to Decrypt() = 10002

Thus I was able to conclude that bucketKeyEnabled works regardless of whether you set at the bucket level or object level. Although, I do not know how it is optimized for both access patterns internally

aws s3 | bucket key enabled

1 Answers