How to generate md5 checksum for AWS S3 multi-part upload?

Question

I am successfully uploading multi-part files to AWS S3, but now I'm attempting to ad an MD5 checksum to each part:

static void sendPart(existingBucketName, keyName, multipartRepsonse, partNum,
                     sendBuffer, partSize, vertx, partETags, s3, req, resultClosure)
{

    // Create request to upload a part.
    MessageDigest md = MessageDigest.getInstance("MD5")
    byte[] digest = md.digest(sendBuffer.bytes)
    println(digest.toString())
    InputStream inputStream = new ByteArrayInputStream(sendBuffer.bytes)
    UploadPartRequest uploadRequest = new UploadPartRequest()
        .withBucketName(existingBucketName).withKey(keyName)
        .withUploadId(multipartRepsonse.getUploadId()).withPartNumber(partNum)
        .withInputStream(inputStream)
        .withMD5Digest(Base64.getEncoder().encode(digest).toString())
        .withPartSize(partSize);

    // Upload part and add response to our list.
    vertx.executeBlocking({ future ->

            // Do the blocking operation in here

            // Imagine this was a call to a blocking API to get the result
            try {
                println("Sending chunk for ${keyName}")
                PartETag eTag = s3.uploadPart(uploadRequest).getPartETag()
                partETags.add(eTag);
                println("Etag: " + eTag.ETag)
                req.response().write("Sending Chunk\n")
            } catch(Exception e) {
            }

            def result = "success!"

            future.complete(result)
        }, resultClosure)
}

However I get the following error:

AmazonS3Exception: The XML you provided was not well-formed or did not validate against our published schema (Service: Amazon S3; Status Code: 400; Error Code: MalformedXML; Request ID: 91542E819781FDFC), S3 Extended Request ID: yQs45H/ozn5+xlxV9lRgCQWwv6gQysT6A4ablq7/Epq06pUzy0qGvMc+YAkJjo/RsHk2dedH+pI=

What am I doing incorrectly?

cmaynard cmaynard · Accepted Answer · 2016-07-29T15:09:10

Looks like I was converting the digest incorrectly.

static void sendPart(existingBucketName, keyName, multipartRepsonse, partNum,
                     sendBuffer, partSize, vertx, partETags, s3, req, resultClosure)
{

    // Create request to upload a part.
    MessageDigest md = MessageDigest.getInstance("MD5")
    byte[] digest = md.digest(sendBuffer.bytes)
    InputStream inputStream = new ByteArrayInputStream(sendBuffer.bytes)
    UploadPartRequest uploadRequest = new UploadPartRequest()
        .withBucketName(existingBucketName).withKey(keyName)
        .withUploadId(multipartRepsonse.getUploadId()).withPartNumber(partNum)
        .withInputStream(inputStream)
        .withMD5Digest(Base64.getEncoder().encodeToString(digest))
        .withPartSize(partSize)

    // Upload part and add response to our list.
    vertx.executeBlocking({ future ->

            try {
                println("Sending chunk for ${keyName}")
                PartETag eTag = s3.uploadPart(uploadRequest).getPartETag()
                partETags.add(eTag);
                req.response().write("Sending Chunk\n")
            } catch(Exception e) {
            }

            def result = "success!"

            future.complete(result)
        }, resultClosure)
}

How to generate md5 checksum for AWS S3 multi-part upload?

1 Answers