BigQuery export data in one big file

1
votes

The BigQuery docs tell us about limits per request, one file can't be more than 1 Gb and it true, but not always. I did a new export and was surprised, a file had a 6 Gb size. And this big file is a problem for me.

What I can do for decrease file size and why BigQuery did so big file?

1
google-bigquerygoogle-cloud-storage
Hi @Artem, Can you tell us how you do the export. In the link, you provided it state: When you export data to multiple files, the size of the files will vary., Maybe this is the source of your problem. – Tamir Klein
@TamirKlein I tried a few ways, first is Google API, second UI, both results are identical. Yes, the size of the files can be different, but not more than 1Gb. The file contains 4M valid JSON lines – Artem Chernov
Have you tried using a wildcard? – Pol Ortiz
@PolOrtiz yes, of course, it's not my first export from BigQuery, I have a lot of tables which I export every day and all works fine, I have a problem only on the one table – Artem Chernov

1 Answers

2
votes

BigQuery export file sizes vary and there are times it will split into a file more than 1GB or split into many small sized files.

There was a public issue filed about this but it only involved documentation change to set the expectation on varying export file sizes

This appears that others are also encountering the issue :

You can try for a new export but if the export file sizes are consistently above 1GB, you may report for a public issue as of the moment to review this unpredictable behavior.