Handling timeouts during gcloud sql instance creation

4
votes

I'm using gcloud sql instances create to create CloudSQL instances. More often than not, that command times out, but provides a command to wait until databse creation is done.

$ gcloud sql instances create mydb
Creating Cloud SQL instance...failed.
ERROR: (gcloud.sql.instances.create) Operation https://www.googleapis.com/sql/v1beta4/projects/foobar/operations/abcd is taking longer than expected. 
You can continue waiting for the operation by running `gcloud sql operations wait --project foobar abcd`
$ gcloud sql operations wait --project foobar abcd
Waiting for [https://www.googleapis.com/sql/v1beta4/projects/foobar/operations/abcd]...done.
NAME  TYPE    START                          END                            ERROR  STATUS
abcd  CREATE  2018-03-19T15:04:29.477+00:00  2018-03-19T15:10:08.561+00:00  -      DONE

I wonder if it's possible to configure the timeout for gcloud sql instances create? I can't seem to find anything in the docs, not even the timeout it's using by default.

Another option would be to start instance creation asynchronously, list the pending operations and wait for their completion:

$ gcloud sql instances create mydb --async
$ PENDING_OPERATIONS=$(gcloud sql operations list --instance=mydb --filter='status!=DONE' --format='value(name)')
$ gcloud sql operations wait "${PENDING_OPERATIONS}"

But then again, would gcloud sql operations wait timeout at some point? The docs don't mention any timeout either.

2
google-cloud-platformgoogle-cloud-sql
I went through the docs and can't find a timeout either. Mind you, the timeout won't affect whether an operation goes through or not. I'm also curious why your operation is timing out, gcloud sql instances create is a pretty simply operation and shouldn't take long to complete - Patrick W
Where are you running the command from? Can you try running the same command from the Cloud shell or another system? - Patrick W
@PatrickW I created 4 instances yesterday, and hit the timeout 3 times. I'm running the command from my workstation in Germany to create instances in region europe-west1. I haven't tried running it from Cloud shell, because the "create instance" command is just one part of a script and the timeout isn't a big issue. - otto.poellath
I just bumped into this error and issued the 'wait' command. It timed out a second time and prompted me again to issue the 'wait' command. Perhaps this is the wrong place for a rant, but WTF? This is amateurish at best. Succeed quietly, fail loudly....don't issue a message that says "ERROR" about an operation that hasn't even completed! This is just pure madness on Google's part. - William Pursell

2 Answers

1
votes

There is a --timeout flag in the wait command (not sure if it's a recent addition). The default is 300s.

https://cloud.google.com/sdk/gcloud/reference/sql/operations/wait

So you can do gcloud sql operations wait "${PENDING_OPERATIONS}" --timeout=unlimited

0
votes

I think it times out because I'm creating the MySQL instance on the smallest machine available: db-f1-micro which is enough for my dev platform.

I've tried several options to capture the command output and run the suggested command, but can't find a way to get it. It's as if the output was printed on something else that stderr or stdout.

I've tried the following :

MyVAR=$($COMMAND)
$COMMAND | tee command.log
$COMMAND 2>&1 > command.log

if we could capture the output of the command, then my script bellow would work.

gcloud sql instances create ${MYSQL_INSTANCE}    \
        --assign-ip                              \
        ${BACKUP}                                \
        ${BACKUP_START_TIME}                     \
        ${BACKUP_ENABLE_BIN_LOG}                 \
        --database-flags=${MYSQL_FLAGS}          \
        --database-version=${MYSQL_DB_VERSION}   \
        --gce-zone=${MYSQL_ZONE}                 \
        --maintenance-release-channel=production \
        --maintenance-window-day=MON             \
        --maintenance-window-hour=4              \
        --pricing-plan=PER_USE                   \
        --storage-auto-increase                  \
        --storage-size=${MYSQL_STORAGE}          \
        --storage-type=${MYSQL_STORAGE_TYPE}     \
        --tier=${MYSQL_TIER}                     \
        --format=json                            | tee logs/${COUNTRY}-${ENV}.log

GCLOUD_CREATE_INSTANCE_RETURN_CODE=$?

if [ ${GCLOUD_CREATE_INSTANCE_RETURN_CODE} -eq 0 ]
then
  echo "Successfully created DB"
else
  echo "Error returned while creating the DB"

  TIMEOUT_ERROR=$(cat logs/${COUNTRY}-${ENV}.log | tail -1 | grep "gcloud beta sql operations wait" | wc -l)

  if [ "${TIMEOUT_ERROR}" = "1" ]
  then
    echo "It's a timeout error, continue waiting"
    FOLLOW_UP_COMMAND=$(cat logs/${COUNTRY}-${ENV}.log | tail -1 |cut -d '`' -f2)
    ${FOLLOW_UP_COMMAND}
  else
    echo "It's NOT a timeout error"
    echo "return code : ${GCLOUD_CREATE_INSTANCE_RETURN_CODE}"
    echo  ${GCLOUD_CREATE_INSTANCE_OUTPUT}
    exit 1
  fi
fi

gcloud sql users set-password root --instance=${MYSQL_INSTANCE} --password=${MYSQL_ROOT} --host "%"

In the end, I ended up putting a big sleep after the create instance command so that I can set the root password.

for db-f1-micro it took 2min53sec more to complete the creation.