Multipart Upload Amazon S3

Question

I'm trying to upload a file on Amazon S3 using their APIs. I tried using their sample code and it creates various parts of files. Now, the problem is, how do I pause the upload and then resume it ? See the following code as given on their documentation:

import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;

import com.amazonaws.auth.profile.ProfileCredentialsProvider;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
import com.amazonaws.services.s3.model.PartETag;
import com.amazonaws.services.s3.model.UploadPartRequest;

public class UploadObjectMPULowLevelAPI {

public static void main(String[] args) throws IOException {
    String existingBucketName  = "*** Provide-Your-Existing-BucketName ***"; 
    String keyName             = "*** Provide-Key-Name ***";
    String filePath            = "*** Provide-File-Path ***";   

    AmazonS3 s3Client = new AmazonS3Client(new ProfileCredentialsProvider());        

    // Create a list of UploadPartResponse objects. You get one of these
    // for each part upload.
    List<PartETag> partETags = new ArrayList<PartETag>();

    // Step 1: Initialize.
    InitiateMultipartUploadRequest initRequest = new 
         InitiateMultipartUploadRequest(existingBucketName, keyName);
    InitiateMultipartUploadResult initResponse = 
                           s3Client.initiateMultipartUpload(initRequest);

    File file = new File(filePath);
    long contentLength = file.length();
    long partSize = 5242880; // Set part size to 5 MB.

    try {
        // Step 2: Upload parts.
        long filePosition = 0;
        for (int i = 1; filePosition < contentLength; i++) {
            // Last part can be less than 5 MB. Adjust part size.
            partSize = Math.min(partSize, (contentLength - filePosition));

            // Create request to upload a part.
            UploadPartRequest uploadRequest = new UploadPartRequest()
                .withBucketName(existingBucketName).withKey(keyName)
                .withUploadId(initResponse.getUploadId()).withPartNumber(i)
                .withFileOffset(filePosition)
                .withFile(file)
                .withPartSize(partSize);

            // Upload part and add response to our list.
            partETags.add(
                    s3Client.uploadPart(uploadRequest).getPartETag());

            filePosition += partSize;
        }

        // Step 3: Complete.
        CompleteMultipartUploadRequest compRequest = new 
                     CompleteMultipartUploadRequest(
                                existingBucketName, 
                                keyName, 
                                initResponse.getUploadId(), 
                                partETags);

        s3Client.completeMultipartUpload(compRequest);
    } 
    catch (Exception e) 
    {
        s3Client.abortMultipartUpload(new AbortMultipartUploadRequest(
                existingBucketName, keyName, initResponse.getUploadId()));
    }
}
}

I have also tried the TransferManager example which takes an Upload object and calls a tryPause(forceCancel) method. But the problem here is, it gets cancelled everytime I try and pause it.

My question is, how do I use the above code with pause and resume functionalities ? Also, just to note that I would also like to upload multiple files with same functionalities.... Help would be much appreciated.

Richard Fung Richard Fung · Accepted Answer · 2014-08-21T00:01:12

I think you should use the Transfer Manager sample if you can. If it's being canceled, it's likely that it just isn't possible to pause it(with the given configuration of the TransferManager you are using).

This might be because you paused it too early to make "pausing" mean anything besides canceling, you are trying to use encryption, or the file isn't big enough. I believe the default minimum file size is 16MB. However, you can change the configuration of the TransferManager to allow you to pause depending on tryPause is failing, except in the case of encryption where I don't think there's anything you can do.

If you want to enable pause/resume for a file smaller than that size, you can call the setMultipartUploadThreshold(long) method in TransferManagerConfiguration. If you want to be able to pause earlier, you can use setMinimumUploadPartSize to set it to use smaller chunks.

In any case, I would advise you to use the TransferManager if possible, since it's made to do this kind of thing for you. It might be helpful to see why the transfer is not being paused when you use tryPause.

Multipart Upload Amazon S3

2 Answers