Loading json format data into google bigquery performance issue

1
votes

I have loaded JSON format data structure into Google bigquery "Nested" table (I have 2 levels of nested "repeated" records ) the average length of JSON line is 5000 characters. The load time is much slower than loading flat file( same size in total ) into Google bigquery .

  1. What are the "rule of thumbs" while loading json into nested records?

  2. How can i improve my performance ?

  3. In terms query of performance, is it much slower also to retreive date from nested table , than flat table ?

Please Help , I have found it difficult to reach experienced "DBA" in that area

Regards

1
jsongoogle-bigquery

1 Answers

0
votes

  1. I don't know of any reason json imports should be slower, but we haven't benchmarked them.

  2. If perf is slow, you may be better off breaking the import into chunks and passing multiple source files into the load job.

  3. It shouldn't be any slower retrieving the data from the nested table (and might be faster). The columnar storage format should store your nested data more efficiently than a corresponding flat table.