“resourcesExceeded” error when creating a table from a .avro file in BigQuery

2
votes

I have uploaded a .avro file on Google Cloud Storage which is about 100MB. It is converted from a 800MB .csv file.

When trying to create a table from this file in the BigQuery web interface, I get the following error after a few seconds:

script: Resources exceeded during query execution: UDF out of memory. (error code: resourcesExceeded)
Job ID  audiboxes:bquijob_4462680b_15607de51b9

I checked the BigQuery Quota Policy and I think my file does not exceed it.

Is there a workaround or do I need to split my original .csv in order to get multiple, smaller .avro files ?

Thanks in advance !

1
csvgoogle-bigquerygoogle-cloud-storageavro

1 Answers

1
votes

This error means that the parser used more memory than allowed. We are working on fixing this issue. In the meantime, if you used compression in the Avro files, try remove it. Using a smaller data block size will also help.

And yes splitting into smaller Avro files like 10MB or less will help too, but the two approaches above are easier if they work for you.