BigQuery table from dataframe

Question

I would like to create a bigQuery table for genomic data with Google Datalab notebook and hitting the columns limit. The table is created from pandas DataFrame.

# to communicate with Google BigQuery
from pandas.io import gbq
gbq.to_gbq(df_pgp_input, 'pgp_orielresearch.pgp_PIK3CA_all_ml', 'speedy-emissary-167213', chunksize=10000, verbose=True)

GenericGBQException: Reason: invalid, Message: Too many fields: 12427

Answer 1

As stated in the docs, the max limit for total columns in Bigquery is 10000.

That's why for the most part it's better if you arrange your data using ARRAYs and STRUCTs as you won't hit these quota limits.