Create table of datastore backup via BigQuery Java api

Question

I have a Google Cloud Datastore backup on Google Cloud Storage and I would like to import it automatically into BigQuery.

TableDefinition tableDefinition = ExternalTableDefinition
        .newBuilder(tableUri, Schema.of(), FormatOptions.datastoreBackup())
        .setAutodetect(true)
        .build();
return bigQuery.create(TableInfo.of(TableId.of("ds", tableName), tableDefinition));

This trows the following exception;

com.google.cloud.bigquery.BigQueryException: Specifying a schema is disallowed for STORAGE_FORMAT_DATASTORE_BACKUP

If I change Schema.of() to null it throws a null pointer. And all factory methods have a method signature which requires a scheme. How can I create this table as an external table via Java API?

John Muraguri John Muraguri · Accepted Answer · 2017-08-29T19:40:58

Try this

public class DatastoreBackupImport {

        private String datasetName = "...";
        private String uri = "gs://xxx/xxx/.xxx.backup_info";
        private String tableName = "xxx";

        private void importBackup(){
            BigQuery bigquery = BigQueryOptions.getDefaultInstance().getService();
            TableId tableId = TableId.of(datasetName, tableName);
            TableDefinition tableDefinition = StandardTableDefinition.newBuilder().build();
            TableInfo tableInfo = TableInfo.newBuilder(tableId, tableDefinition).build();
            Table table = bigquery.create(tableInfo);
            Job job = table.load(FormatOptions.datastoreBackup(), uri);
    // Wait for the job to complete
            try {
                Job completedJob = job.waitFor(WaitForOption.checkEvery(1, TimeUnit.SECONDS),
                    WaitForOption.timeout(3, TimeUnit.MINUTES));
                if (completedJob != null && completedJob.getStatus().getError() == null) {
                    // Job completed successfully
                    System.out.println("Job completed successfully");
                } else {
                    // Handle error case
                    System.out.printf("There was an error");
                    System.out.println(completedJob.getStatus().getError().getMessage());
                }
            } catch (InterruptedException | TimeoutException e) {
                // Handle interrupted wait
                System.out.println("There was an interruption");
            }
        }
    }

Create table of datastore backup via BigQuery Java api

1 Answers