I am trying to upload gzip compressed files using Google's BigQuery Java client API. I am able to upload normal files without any issue. But gzip fails with the error "Invalid content type 'application/x-gzip'. Uploads must have content type 'application/octet-stream'".
Below is my code.
val pid = "****"
val dsid = "****"
val tid = "****"
val br = Source.fromFile(new File("****")).bufferedReader()
val mapper = new ObjectMapper()
val schemaFields = mapper.readValue(br, classOf[util.ArrayList[TableFieldSchema]])
val tschema = new TableSchema().setFields(schemaFields)
val tr = new TableReference().setProjectId(pid).setDatasetId(dsid).setTableId(tid)
val jc = new JobConfigurationLoad().setDestinationTable(tr)
.setSchema(tschema)
.setSourceFormat("NEWLINE_DELIMITED_JSON")
.setCreateDisposition("CREATE_IF_NEEDED")
.setWriteDisposition("WRITE_APPEND")
.setIgnoreUnknownValues(true)
val fmr = new SimpleDateFormat("dd-MM-yyyy_HH-mm-ss-SSS")
val now = fmr.format(new Date())
val loadJob = new Job().setJobReference(new JobReference().setJobId(Joiner.on("-")
.join("INSERT", pid, dsid, tid, now))
.setProjectId(pid))
.setConfiguration(new JobConfiguration().setLoad(jc))
// val data = new FileContent(MediaType.OCTET_STREAM.toString, new File("/Users/jegan/sessions/34560-6")) // This works.
val data = new FileContent(MediaType.GZIP.toString, new File("/Users/jegan/sessions/34560-6"))
val bq = BQHelper.createAuthorizedClientWithDefaultCredentials()
val job = bq.jobs().insert(pid, loadJob, data).execute()
And from this link, I see that we need to use resumable upload to achieve this.
https://cloud.google.com/bigquery/loading-data-post-request#resumable
But the issue is, I am using the Java Client library from Google. How to do resumable upload using this library? There seems to be not much information on this regard or I am missing something. Has anyone ever done this? Please point me to some documentation/samples. Thanks.
