How to use boto's s3 bucket.copy_key with KMS SSE?

Question

I am using boto version 2.38.0 in an attempt to do a region-to-region copy of a key on s3 to the same key in a different bucket that is in a different region. The source key is encrypted using a KMS key, and the destination should also be encrypted using a KMS key, albeit a different one.

I can properly copy to an unencrypted destination key using the following code:

import boto
from boto import connect_s3
if not boto.config.get('s3', 'use-sigv4'):
    boto.config.add_section('s3')
    boto.config.set('s3', 'use-sigv4', 'True')
src_bucket_name = 'my-bucket'
dest_bucket_name = 'my-bucket-2'
conn_std = connect_s3(host='s3.amazonaws.com')
conn_norcal = connect_s3(host='s3-us-west-1.amazonaws.com')
src = conn_std.get_bucket(src_bucket_name)
dest = conn_norcal.get_bucket(dest_bucket_name)
for src_item in src_items:
    dest.copy_key(src_item.key, src.name, src_item.key)

However, if I attempt to enable SSE with KMS key using the headers specified here (http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectCOPY.html), I get a SignatureDoesNotMatch error.

So, for example, if I change the copy_key line to something like:

dest.copy_key(src_item.key, src.name, src_item.key, headers={
    'x-amz-server-side-encryption':'aws:kms',
    'x-amz-server-side-encryption-aws-kms-key-id': 'e2009179-a4fc-4a0a-99c6-490b7bb1ebbb'
    })

Then I get an error like:

## Note: I replaced all sensitive details with [...], 
## and used my-bucket and my-bucket-2 with testing.txt.

Traceback (most recent call last):
[...]
  File "/Library/Python/2.7/site-packages/boto/s3/bucket.py", line 888, in copy_key
    response.reason, body)
boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>[...]</AWSAccessKeyId><StringToSign>AWS4-HMAC-SHA256
20150818T231356Z
20150818/us-west-1/s3/aws4_request
e2466b937a1a45c13fa055db0df7b89476ff568f2beef309743ad937b351ce30</StringToSign><SignatureProvided>7df5b18a8015c325e9363c84c8a919e60392f3b7ef20e48b70c45eb0785a7431</SignatureProvided><StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 31 35 30 38 31 38 54 32 33 31 33 35 36 5a 0a 32 30 31 35 30 38 31 38 2f 75 73 2d 77 65 73 74 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 65 32 34 36 36 62 39 33 37 61 31 61 34 35 63 31 33 66 61 30 35 35 64 62 30 64 66 37 62 38 39 34 37 36 66 66 35 36 38 66 32 62 65 65 66 33 30 39 37 34 33 61 64 39 33 37 62 33 35 31 63 65 33 30</StringToSignBytes><CanonicalRequest>PUT
/testing.txt

host:my-bucket-2.s3-us-west-1.amazonaws.com
user-agent:Boto/2.38.0 Python/2.7.6 Darwin/14.3.0
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-copy-source:my-bucket/testing.txt
x-amz-date:20150818T231356Z
x-amz-metadata-directive:COPY
x-amz-server-side-encryption:aws:kms
x-amz-server-side-encryption-aws-kms-key-id:e2009179-a4fc-4a0a-99c6-490b7bb1ebbb
x-amz-storage-class:STANDARD

host;user-agent;x-amz-content-sha256;x-amz-copy-source;x-amz-date;x-amz-metadata-directive;x-amz-server-side-encryption;x-amz-server-side-encryption-aws-kms-key-id;x-amz-storage-class
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855</CanonicalRequest><CanonicalRequestBytes>50 55 54 0a 2f 61 65 74 6e 61 2f 6d 65 74 61 64 61 74 61 2f 61 65 74 6e 61 5f 63 6f 6c 75 6d 6e 5f 64 65 74 61 69 6c 73 2e 74 78 74 0a 0a 68 6f 73 74 3a 71 75 61 72 74 65 74 68 65 61 6c 74 68 2d 73 65 63 75 72 65 2d 62 61 63 6b 75 70 2e 73 33 2d 75 73 2d 77 65 73 74 2d 31 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 75 73 65 72 2d 61 67 65 6e 74 3a 42 6f 74 6f 2f 32 2e 33 38 2e 30 20 50 79 74 68 6f 6e 2f 32 2e 37 2e 36 20 44 61 72 77 69 6e 2f 31 34 2e 33 2e 30 0a 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35 0a 78 2d 61 6d 7a 2d 63 6f 70 79 2d 73 6f 75 72 63 65 3a 71 75 61 72 74 65 74 68 65 61 6c 74 68 2d 73 65 63 75 72 65 2f 61 65 74 6e 61 2f 6d 65 74 61 64 61 74 61 2f 61 65 74 6e 61 5f 63 6f 6c 75 6d 6e 5f 64 65 74 61 69 6c 73 2e 74 78 74 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 32 30 31 35 30 38 31 38 54 32 33 31 33 35 36 5a 0a 78 2d 61 6d 7a 2d 6d 65 74 61 64 61 74 61 2d 64 69 72 65 63 74 69 76 65 3a 43 4f 50 59 0a 78 2d 61 6d 7a 2d 73 65 72 76 65 72 2d 73 69 64 65 2d 65 6e 63 72 79 70 74 69 6f 6e 3a 61 77 73 3a 6b 6d 73 0a 78 2d 61 6d 7a 2d 73 65 72 76 65 72 2d 73 69 64 65 2d 65 6e 63 72 79 70 74 69 6f 6e 2d 61 77 73 2d 6b 6d 73 2d 6b 65 79 2d 69 64 3a 65 32 30 30 39 31 37 39 2d 61 34 66 63 2d 34 61 30 61 2d 39 39 63 36 2d 34 39 30 62 37 62 62 31 65 33 62 32 0a 78 2d 61 6d 7a 2d 73 74 6f 72 61 67 65 2d 63 6c 61 73 73 3a 53 54 41 4e 44 41 52 44 0a 0a 68 6f 73 74 3b 75 73 65 72 2d 61 67 65 6e 74 3b 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3b 78 2d 61 6d 7a 2d 63 6f 70 79 2d 73 6f 75 72 63 65 3b 78 2d 61 6d 7a 2d 64 61 74 65 3b 78 2d 61 6d 7a 2d 6d 65 74 61 64 61 74 61 2d 64 69 72 65 63 74 69 76 65 3b 78 2d 61 6d 7a 2d 73 65 72 76 65 72 2d 73 69 64 65 2d 65 6e 63 72 79 70 74 69 6f 6e 3b 78 2d 61 6d 7a 2d 73 65 72 76 65 72 2d 73 69 64 65 2d 65 6e 63 72 79 70 74 69 6f 6e 2d 61 77 73 2d 6b 6d 73 2d 6b 65 79 2d 69 64 3b 78 2d 61 6d 7a 2d 73 74 6f 72 61 67 65 2d 63 6c 61 73 73 0a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35</CanonicalRequestBytes><RequestId>D0092E395ACDBBBB</RequestId><HostId>jnUTMiF5ohPZBWhz9EboX66iym/9nFQUMUEB2Nk/LsuYiSorG5z3sxkrqIQIPt+1DDhCZxuBvUg=</HostId></Error>

I looked through the "Troubleshooting AWS Signature Version 4 Errors" page (http://docs.aws.amazon.com/general/latest/gr/signature-v4-troubleshooting.html), which maybe can give some hints, but that appears to be helpful in troubleshooting one's own custom signing code, not the boto SDK's.

Am I doing something wrong? How can properly copy a key from one region to another (with no local intermediary) while also specifying a KMS encryption key to use for Server Side Encryption?

Alec Alec · Accepted Answer · 2015-08-19T16:07:23

Looks like this is indeed a bug in boto, where the headers are improperly sorted in the canonical representation. They end up getting sorted as fully 'key:value' instead of just sorting by 'key', which ends up producing different results with certain sets of keys.

Unfortunately, these PRs are still outstanding:

https://github.com/boto/boto/pull/3176

https://github.com/boto/boto/pull/3032

I reimplemented the upload in boto3, and found that the code works properly there:

import boto3

# note, be sure to put '[s3] use-sigv4 = True' in ~/.boto

src_bucket_name = 'my-bucket'
dest_bucket_name = 'my-bucket-2'
test_key = '/testing.txt'
kms_key = '1e7ff509-6f87-412f-a7af-07ff72ea1111'
dest_bucket_region = 'us-west-1'

client = boto3.client('s3', dest_bucket_region)
response = client.copy_object(
    Bucket=dest_bucket_name,
    CopySource=src_bucket_name+test_key, # apparently must not be urlencoded, contrary to documentation
    Key=test_key,
    ServerSideEncryption='aws:kms',
    StorageClass='STANDARD',
    SSEKMSKeyId=kms_key
    )

Guess it pays to use the latest and greatest!

How to use boto's s3 bucket.copy_key with KMS SSE?

1 Answers