S3 ETag for an 110MB file not MD5

Question

My understanding of ETag was that it should correspond to MD5 hash provided that you haven't used multipart upload and you're not using KMS encryption.

I just started depending on this functionality in an application and have found that this doesn't hold.

I've uploaded a largeish powerpoint file (110MB) via cli and the ETag is not an MD5 hash.

Used precisely the same cli command with a smaller powerpoint file and the ETag is an MD5 hash.

Same behaviour is observed via the console. Are cli and console using multipart upload under the hood or is there something else I missed?


jgreenal$ aws s3 cp bigpowerpoint.pptx s3://mybucketname --sse=AES256

jgreenal$ aws s3api head-object --bucket mybucketname --key bigpowerpoint.pptx
{
    "AcceptRanges": "bytes",
    "LastModified": "Fri, 06 Mar 2020 11:15:47 GMT",
    "ContentLength": 113962008,
    "ETag": "\"d15e9a175cd9365a6480eab9105c7c86-14\"",
    "VersionId": "2JMq2pPyKMk3ZlSco4l.r5ZPip_pp6Oq",
    "ContentType": "application/vnd.openxmlformats-officedocument.presentationml.presentation",
    "ServerSideEncryption": "AES256",
    "Metadata": {},
    "ReplicationStatus": "COMPLETED"
}

jgreenal$ aws s3 cp tinypowerpoint.pptx s3://mybucketname --sse=AES256

jgreenal$ aws s3api head-object --bucket mybucketname --key tinypowerpoint.pptx
{
    "AcceptRanges": "bytes",
    "LastModified": "Fri, 06 Mar 2020 11:20:37 GMT",
    "ContentLength": 33878,
    "ETag": "\"7a0b3c27805d5f599e5d655b13b4644c\"",
    "VersionId": "Tv1WyDSoHo2s.p59IKM_M6iaVGdh6EiF",
    "ContentType": "application/vnd.openxmlformats-officedocument.presentationml.presentation",
    "ServerSideEncryption": "AES256",
    "Metadata": {},
    "ReplicationStatus": "COMPLETED"
}

John Greenall John Greenall · Accepted Answer · 2020-03-06T11:49:18

Sorry - the answer to this was already in my question - should have checked deeper first.

aws s3 cp does indeed use multipart upload under the hood.

jgreenal$ aws s3api put-object  --bucket my-bucketname --key bigpowerpointput.pptx --body bigpowerpoint.pptx --server-side-encryption AES256
jgreenal$ aws s3api head-object --bucket my-bucketname --key bigpowerpointput.pptx
{
    "AcceptRanges": "bytes",
    "LastModified": "Fri, 06 Mar 2020 11:43:21 GMT",
    "ContentLength": 113962008,
    "ETag": "\"a11619dc992b584587c24668c767a46c\"",
    "VersionId": "DYG.5IOkbJncwCT6iOK5c2seKBtDTg_7",
    "ContentType": "binary/octet-stream",
    "ServerSideEncryption": "AES256",
    "Metadata": {},
    "ReplicationStatus": "COMPLETED"
}

S3 ETag for an 110MB file not MD5

1 Answers