Querying compressed files using BigQuery federated source

2
votes

According to the BigQuery federated source documentation:

[...]or are compressed must be less than 1 GB each.

This would imply that compressed files are supported types for federated sources in BigQuery.

However, I get the following error when trying to query a gz file in GCS:

enter image description here

I tested with an uncompressed file and it works fine. Are compressed files supported as federated sources in BigQuery, or have I misinterpreted the documentation?

1
google-bigquerygoogle-cloud-storage
You certainly shouldn't be getting an internal error (although there might be something else wrong). I'll file a bug with the team and get back to you when I have some additional information. – Adam Lydick

1 Answers

2
votes

Compression mode defaults to NONE and needs to be explicitly specified in the external table definition.

At the time of the question, this couldn't be done through the UI. This is now fixed and compressed data should be automatically detected.

For more background information, see: https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.query

The interesting parameter is "configuration.query.tableDefinitions.[key].compression".