Apache Beam - CassandraIO write Async - 2.6.0 error

2
votes

I am using following libraries for apache beam to run dataflow job to read data from BigQuery and Store/Write to Cassandra.

  1. beam-sdks-java-io-cassandra - 2.6.0
  2. beam-sdks-java-io-jdbc - 2.6.0
  3. beam-sdks-java-io-google-cloud-platform - 2.6.0
  4. beam-sdks-java-core - 2.6.0
  5. google-cloud-dataflow-java-sdk-all - 2.5.0
  6. google-api-client -1.25.0

Since beam-sdks-java-io-cassandra > 2.3 version support saveAsync, I have upgraded all my libraries to 2.6.0.

After libraries update, I am getting following error at the time insert/save data to Cassandra. 

java.lang.NoSuchMethodError: com.datastax.driver.mapping.Mapper.saveAsync(Ljava/lang/Object;)Lorg/apache/beam/repackaged/beam_sdks_java_io_cassandra/com/google/common/util/concurrent/ListenableFuture;
    at org.apache.beam.sdk.io.cassandra.CassandraServiceImpl$WriterImpl.write(CassandraServiceImpl.java:435)
    at org.apache.beam.sdk.io.cassandra.CassandraIO$WriteFn.processElement(CassandraIO.java:493)

It looks like it is an issue of ListenableFuture from Gauva and Cassandra Driver.

1
google-cloud-dataflowapache-beam
I just faced the same exception. Did you find a workaround? - Maksim Kolchin
@MaximKolchin please see my answer below. - srfrnk
thanks a lot. The apache Beam has fixed this issue in 2.11.0 issues.apache.org/jira/browse/… - ASe

1 Answers

2
votes

I have a workaround for this. Use beam-sdks-java-io-cassandra - 2.4.0. I'm working on fixing this incl. some other stuff... will update here.

Update: Most likely found the issue. Pushed a fix to my own fork. However it might take sometime until this can be made into a PR and released by the guys at Beam... If anyone wants to use the version I built - you can take a look at how it's done here