Getting “Backend error. Job aborted” while trying to export a big query table to GCS

0
votes

Since the past couple of weeks I have been continuously getting "Backend error. Job aborted" error while trying to export a big query table to google cloud storage in csv format.

The table has been created using bq select * statement (using allowLargeQueryResult option)

Also the target bucket name doesn't seem to be problematic.

Here's a sample extract.

Errors: Backend error. Job aborted. Job ID: kiwiup.com:kiwi-bigquery:job_mk90xJqtyinbzRqIfWVjM2mHLP0 Start Time: 2:53pm, 8 Aug 2014 End Time: 8:53pm, 8 Aug 2014

The job is taking almost six hours to complete after which it fails. Previously it used to complete in a couple of minutes. Any help would be appreciated.

2
google-bigquery
Can you send a job id of a failed job? Thanks - Jordan Tigani
edited in the question. Thanks - ferrisbueller
Hi Jordan, Still waiting for an answer to this one. It is critical for our data migration activity. - ferrisbueller

2 Answers

0
votes

Your export job hit a timeout. We're currently investigating why; the date of your job coincides with a bandwidth issue we were having that should have been resolved. We're currently adding more instrumentation and monitoring so it will be easier to debug in the future.

As a workaround, if you give multiple extraction URI patterns, BigQuery will spin up more workers in parallel. See the "Multiple Wildcard URIs" example here.

0
votes

As Jordan said, this coincided with a bandwidth problem. Sorry for the inconvenience.

In some cases, giving multiple wildcard URIs will increase parallelism, but this applies only to fairly large (10's of GB) tables, and can actually decrease parallelism. Multiple wildcard URIs are designed to support Hadoop jobs, not to control parallelism.