Apache beam and BigQuery

Question

I'm trying to execute apache beam sdk 2.4 and the libraries com.google.cloud.bigquery but thow exception

Exception in thread main java.lang.NoSuchMethodError com.google.api.client.googleapis.services.json.AbstractGoogleJsonClient$Builder.setBatchPath(LjavalangString;)LcomgoogleapiclientgoogleapisservicesAbstractGoogleClient$Builder;
at com.google.api.services.bigquery.Bigquery$Builder.setBatchPath(Bigquery.java3519)

import com.google.cloud.bigquery.*;

<dependency>
    <groupId>com.google.cloud.dataflow</groupId>
    <artifactId>google-cloud-dataflow-java-sdk-all</artifactId>
    <version>[2.4.0, 2.99)</version>
</dependency>

<!-- slf4j API frontend binding with JUL backend -->
<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-api</artifactId>
    <versi

on>1.7.14</version>
    </dependency>

   <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-jdk14</artifactId>
        <version>1.7.14</version>
    </dependency>

    <dependency>
        <groupId>com.google.cloud</groupId>
        <artifactId>google-cloud-bigquery</artifactId>
        <version>1.35.0</version>
    </dependency>

</dependencies>

the code example apache beam 2.4 using

import com.google.api.services.bigquery.model.TableFieldSchema;
import com.google.api.services.bigquery.model.TableRow;
import com.google.api.services.bigquery.model.TableSchema;

If I use the the libraries om.google.api.services.bigquery. the code example run successfully .

What library is it using internally?

What library should I use? com.google.api.services.bigquery or com.google.cloud.bigquery?

Andrew Pilloud Andrew Pilloud · Accepted Answer · 2018-06-28T17:01:19

Beam 2.4 is using com.google.api.services.bigquery v2-rev374-1.22.0. This is also the library that you should use with Beam 2.4. The com.google.api.services.bigquery library is out of date and there is an open issue to upgrade it. Unfortunately other components in Beam depend on libraries which are incompatible with com.google.cloud.bigquery, and those need to be upgraded before the new library can be used.

If you aren't already doing so, you should take a look at the BigQuery IO, which provides a wrapper to common BigQuery operations for Beam.

Apache beam and BigQuery

1 Answers