Getting Issue with BigQuery

Question

I am running a sample Java code where I am trying to get the description of BigQuery table. When I run it in my local, its working fine. But when I go to google cloud and deploy the jar in GCS and run it via dataproc, I'm getting below issue

Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;  
    at com.google.api.gax.retrying.BasicRetryingFuture.<init>(BasicRetryingFuture.java:88)  
    at com.google.api.gax.retrying.DirectRetryingExecutor.createFuture(DirectRetryingExecutor.java:88)  
    at com.google.api.gax.retrying.DirectRetryingExecutor.createFuture(DirectRetryingExecutor.java:74)  
    at com.google.cloud.RetryHelper.run(RetryHelper.java:75)  
    at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)  
    at com.google.cloud.bigquery.BigQueryImpl.getTable(BigQueryImpl.java:665)

Sample Code:

private static BigQuery bigquery = null;
    static {
        bigquery = BigQueryOptions.newBuilder()
                .setProjectId("**PROJECT_ID**")
                .build().getService();
    }
 public static void getTable(String projectId, String datasetName, String tableName) {
        try {
           TableId tableId = TableId.of(projectId, datasetName, tableName);
            Table table = bigquery.getTable(tableId);
            System.out.println("Table info: " + table.getDescription());
        } catch (BigQueryException e) {
            System.out.println("Table not retrieved. \n" + e.toString());
        }
    }

pom.xml - This is the only dependency I have in my pom.xml. I have seen suggestions from earlier pos to check the guava dependency conflict, but in my case i don't see a conflict as I don't have the other dependency which can conflict with this.

<dependency>
            <groupId>com.google.cloud</groupId>
            <artifactId>google-cloud-bigquery</artifactId>
            <version>1.116.10</version>
</dependency>

In the Dataproc ui, I am submitting this job by passing jobtype as Spark.

Answer 1

In general, a java.lang.NoSuchMethodError usually indicates that two different versions of an API are in use, and the compiled code expects a different version than the one actually present at the deployment site. In this case, it seems that com.google.common.util.concurrent.MoreExecutors is the conflicting dependency. IIRC it is part of Google Guava.

Check the version history of guava if the method MoreExecutors.directExecutor() has been modified lately, and compare the deployed guava version to your local one.

Answer 2

This issue is resolved. There was a conflict in guava library

<dependency>
     <groupId>com.google.cloud</groupId>
      <artifactId>google-cloud-shared-dependencies</artifactId>
      <version>0.13.0</version>
      <type>pom</type>
      <scope>import</scope>
</dependency>
<dependency>
       <groupId>com.google.cloud</groupId>
       <artifactId>google-cloud-bigquery</artifactId>
       <version>1.116.10</version>
       <exclusions>
           <exclusion>
               <groupId>com.google.guava</groupId>
               <artifactId>guava-jdk5</artifactId>
           </exclusion>
           <exclusion>
               <groupId>com.google.guava</groupId>
               <artifactId>guava</artifactId>
           </exclusion>
           <exclusion>
               <groupId>com.google.guava</groupId>
               <artifactId>failureaccess</artifactId>
           </exclusion>
           <exclusion>
               <groupId>com.google.guava</groupId>
               <artifactId>listenablefuture</artifactId>
           </exclusion>
       </exclusions>
</dependency>
<dependency>
    <groupId>com.google.guava</groupId>
    <artifactId>guava</artifactId>
    <version>30.0-jre</version>
</dependency>

And relocated the com.google.com package to shaded

<relocations>
  <relocation>
    <pattern>com.google.common</pattern>                               
    <shadedPattern>shaded.com.google.common</shadedPattern>
  </relocation>
</relocations>