Is "bigtable-hbase-beam" compatible with beam java SDK 2.5?

Question

I'm following this documentation (specifically, Writing to Cloud Bigtable ): https://cloud.google.com/bigtable/docs/hbase-dataflow-java (I'm trying both versions 1.3.0 and 1.4.0.)

Before running a pipeline, this step gives an error:

CloudBigtableScanConfiguration config = new CloudBigtableScanConfiguration.Builder()
    .withProjectId("project-id")
    .withInstanceId("instance-id")
    .withTableId("table")
    .build();

(Note that the documentation does not state how this 'config' instance is created for the 'writing' section, but I tried both the above (copied from the 'reading' section followed by '.withScan()' line removed as well as CloudBigtableTableConfiguration.Builder instead of ScanConfig. The results are the same on my end.)

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/beam/sdk/repackaged/com/google/common/base/Preconditions
        at com.google.cloud.bigtable.beam.CloudBigtableConfiguration.setValue(CloudBigtableConfiguration.java:142)
        at com.google.cloud.bigtable.beam.CloudBigtableConfiguration.<init>(CloudBigtableConfiguration.java:137)
        at com.google.cloud.bigtable.beam.CloudBigtableTableConfiguration.<init>(CloudBigtableTableConfiguration.java:116)
        at com.google.cloud.bigtable.beam.CloudBigtableScanConfiguration.<init>(CloudBigtableScanConfiguration.java:178)
        at com.google.cloud.bigtable.beam.CloudBigtableScanConfiguration$Builder.build(CloudBigtableScanConfiguration.java:160)
        at com.moloco.dataflow2.Main.main(Main.java:60)
Caused by: java.lang.ClassNotFoundException: org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 7 more

It appears that Preconditions is not found. https://github.com/GoogleCloudPlatform/cloud-bigtable-client/blob/c87f8aef59320f240697fb015dbbc7142a2a679f/bigtable-dataflow-parent/bigtable-hbase-beam/src/main/java/com/google/cloud/bigtable/beam/CloudBigtableConfiguration.java#L143

We've had issues with Beam SDKs 2.0/2.1/2.4, so we are trying to migrate from 1.9.1 to 2.5, and the Bigtable connector is an important feature we utilize a lot.

Which Beam SDKs (java) is bigtable-hbase-beam 1.3.0 compatible with, if not with 2.5 yet?

Among other things, we used the following:

compile group: 'com.google.cloud.bigtable', name: 'bigtable-hbase-beam', version: '1.3.0'
compile 'com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:2.5.0'

Also tried compile 'org.apache.beam:beam-runners-google-cloud-dataflow-java:2.5.0', but with the same error.

Update: Gradle dependency tree suggests that the connector sdk requires beam 2.3+, so I tried 2.3 & 2.4. Unfortunately, it conflicts with other parts of our code and a known issue seems to be blocking us (DataFlow Runner Fails after upgrading to Beam 2.4.0 ). Originally I asked about the version 1.3.0 (not knowing that 1.4.0 is available); but I still get the same error with hbase-bigtable 1.4.0 and beam sdk 2.5.

I've encountered a similar situation with the Cassandra JAR. Did anyone manage to solve this before? Posted here: stackoverflow.com/questions/55254740/… — srfrnk

Solomon Duskis Solomon Duskis · Accepted Answer · 2018-07-09T13:24:36

2

votes

This is a bug in bigtable-hbase-beam. Here's the proposed fix.

Is "bigtable-hbase-beam" compatible with beam java SDK 2.5?

1 Answers