BigQuery: Load Data into EU Dataset from GCS

Question

In the past I have successfully loaded data into US-hosted BigQuery datasets from CSV data in US-hosted GCS buckets. We since decided to move our BigQuery data to the EU and I created a new dataset with this region selected on it. I have successfully populated those of our tables small enough to be uploaded from my machine at home. But two tables are far too large for this so I would like to load them from files in GCS. I have tried doing this from both a US-hosted GCS bucket and an EU-hosted GCS bucket (thinking that bq load might not like to cross regions) but the load fails every time. Below is the error detail I'm getting from the bq command line (500, Internal Error). Does anyone know a reason why this might be happening? Is loading data into EU-hosted BigQuery datasets from GCS something that is known to work for others?

{
  "configuration": {
    "load": {
      "destinationTable": {
        "datasetId": "######", 
        "projectId": "######", 
        "tableId": "test"
      }, 
      "schema": {
        "fields": [
          {
            "name": "test_col", 
            "type": "INTEGER"
          }
        ]
      }, 
      "sourceFormat": "CSV", 
      "sourceUris": [
        "gs://######/test.csv"
      ]
    }
  }, 
  "etag": "######", 
  "id": "######", 
  "jobReference": {
    "jobId": "job_Y4U58uTyxitsvbgljFi2x534N7M", 
    "projectId": "######"
  }, 
  "kind": "bigquery#job", 
  "selfLink": "https://www.googleapis.com/bigquery/v2/projects/######", 
  "statistics": {
    "creationTime": "1445336673213", 
    "endTime": "1445336674738", 
    "startTime": "1445336674738"
  }, 
  "status": {
    "errorResult": {
      "message": "An internal error occurred and the request could not be completed.", 
      "reason": "internalError"
    }, 
    "errors": [
      {
        "message": "An internal error occurred and the request could not be completed.", 
        "reason": "internalError"
      }
    ], 
    "state": "DONE"
  }, 
  "user_email": "######"
}

You have a low rate. Important on SO, you have to mark accepted answers by using the tick on the left of the posted answer, below the voting. This will increase your rate. See how this works by visinting this link: meta.stackoverflow.com/questions/5234/… — Pentium10

Dzamo Norton Dzamo Norton · Accepted Answer · 2015-10-26T06:42:19

After searching through other related questions on StackOverflow I eventually realised that I had set my GCS bucket region to EUROPE-WEST-1 and not the multi-region EU location. Things are now working as expected.

BigQuery: Load Data into EU Dataset from GCS

1 Answers