Cloud Storage Transfer Service from S3 stuck in "this transfer is starting"

Question

A part of our application initiates transfers from Amazon S3 to Google Cloud Storage via the storage transfer service API. We've had this up and running successfully for several months until yesterday when our transfers stopped working. We can see a transfer was initiated in the console, but it hangs indefinitely with a single history item stating: "This transfer is starting..." We have a background process polling the transfer status which we see returning a status of "transfer_calculating"

In trying to debug this issue, we setup a transfer via the storage console. We used the same AWS access key id/secret access key used by our application, and the transfer completed successfully. This leads us to believe the issue is isolated to the transfer service API or our code that initiates the API call.

Transfer Job Code:

TransferJob tjob = new TransferJob()
    .setDescription(description)
    .setStatus('ENABLED')
    .setProjectId(transferGoogleProject)
    .setTransferSpec(
    new TransferSpec()
        .setGcsDataSink(new GcsData().setBucketName(googleStorageBucket))
        .setAwsS3DataSource(
        new AwsS3Data()
            .setBucketName(s3Bucket)
            .setAwsAccessKey(new AwsAccessKey().setAccessKeyId(transferAwsKey).setSecretAccessKey(transferAwsSecret)))
        .setObjectConditions(new ObjectConditions().setIncludePrefixes(s3Keys))
        .setTransferOptions(
        new TransferOptions()
            .setDeleteObjectsFromSourceAfterTransfer(false)
            .setOverwriteObjectsAlreadyExistingInSink(true)
            .setDeleteObjectsUniqueInSink(false)))
    .setSchedule(
    new Schedule()
        .setScheduleStartDate(date)
        .setScheduleEndDate(date)
        .setStartTimeOfDay(time))

tjob = storagetransfer.transferJobs().create(tjob).execute()

Library configuration:

<dependency>
    <groupId>com.google.api-client</groupId>
    <artifactId>google-api-client</artifactId>
    <version>1.19.1</version>
</dependency>
<dependency>
    <groupId>com.google.apis</groupId>
    <artifactId>google-api-services-bigquery</artifactId>
    <version>v2-rev191-1.19.1</version>
</dependency>
<dependency>
    <groupId>com.google.apis</groupId>
    <artifactId>google-api-services-storage</artifactId>
    <version>v1-rev26-1.19.1</version>
</dependency>
<dependency>
    <groupId>com.google.apis</groupId>
    <artifactId>google-api-services-storagetransfer</artifactId>
    <version>v1-rev3-1.19.1</version>
</dependency>
<dependency>
    <groupId>com.google.oauth-client</groupId>
    <artifactId>google-oauth-client</artifactId>
    <version>1.19.0</version>
</dependency>
<dependency>
    <groupId>com.google.http-client</groupId>
    <artifactId>google-http-client</artifactId>
    <version>1.19.0</version>
</dependency>
<dependency>
    <groupId>com.google.http-client</groupId>
    <artifactId>google-http-client-jackson2</artifactId>
    <version>1.19.0</version>
</dependency>

We've bumped the versions up 1.21.0 in our development environment but the transfers still get stuck at "This transfer is starting..."

At this point we're stuck, anyone else running into this issue?

We apologize for the difficulty and are investigating this issue. Could you please send your project IDs and exact queries to: [email protected] so we can look into your exact case? Thanks! -Mayur — Mayur Deshpande

Chris Alme Chris Alme · Accepted Answer · 2016-03-16T16:55:50

From feedback provided by @mayur-deshpande at Google (thanks!), our issue stems from the time value passed to setStartTimeOfDay() needing to be in UTC. Up to this point, we've used US/Pacific which is prescribed in the following snippet from the creating transfers development guide

/**
 * Specify times below using US Pacific Time Zone.
 */
private static final String START_DATE = "YYYY-MM-DD";
private static final String START_TIME = "HH:MM:SS";

Due to the time difference, the time we sent in our request had already passed, so the transfer sat in started status until our time was hit the next day. We saw this happen as the requests did eventually complete.

The javadoc for setStartTime() of day does point out the need to use UTC:

/**
 * The time in UTC at which the transfer will be scheduled to start in a day. Transfers may start
 * later than this time. If not specified, transfers are scheduled to start at midnight UTC.
 * @param startTimeOfDay startTimeOfDay or {@code null} for none
 */
public Schedule setStartTimeOfDay(TimeOfDay startTimeOfDay)

The example code referenced above should reflect this requirement, so I'll file an issue in the github repo.

Also, since we are only doing one time transfers, we made sure to set our start time one minute in the future to account for subtle time differences between servers:

DateTime now = new DateTime().plusMinutes(1)
Date date = new Date().setDay(now.dayOfMonth).setMonth(now.monthOfYear).setYear(now.year)
TimeOfDay time = new TimeOfDay().setHours(now.hourOfDay).setMinutes(now.minuteOfHour).setSeconds(0)

Cloud Storage Transfer Service from S3 stuck in "this transfer is starting"

4 Answers