Unable to use dataflow with google cloud nature language API

Question

I have a dataflow pipepline to use google cloud natural language API for sentimental analysis. So, i include jar for cloud language & dataflow runner as below:

<dependency>
  <groupId>com.google.cloud</groupId>
  <artifactId>google-cloud-language</artifactId>
  <version>1.25.0</version>
</dependency>

<dependency>
    <groupId>org.apache.beam</groupId>
    <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
    <version>2.4.0</version>
</dependency>

But the problem is, there are crashes on the dependencies

cloud language is using io.grpc 1.10.1 while beam sdk is using 1.2.0. Is there any way to sort it out? Thanks.

com.google.cloud:google-cloud-language:jar:1.25.0 -> io.grpc:grpc-stub:jar:1.10.1 -> io.grpc:grpc-core:jar:1.10.1

org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.4.0 -> io.grpc:grpc-core:jar:1.2.0,

Details logs:

[ERROR] Failed to execute goal on project apache-beam-loader: Could not resolve dependencies for project com.sample.cloud:apache-beam-loader:jar:0.0.1-SNAPSHOT: Failed to collect dependencies for com.sample.cloud:apache-beam-loader:jar:0.0.1-SNAPSHOT: Could not resolve version conflict among [org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.4.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.4.0 -> io.grpc:grpc-core:jar:1.2.0, org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.4.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.4.0 -> io.grpc:grpc-netty:jar:1.2.0 -> io.grpc:grpc-core:jar:[1.2.0,1.2.0], org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.4.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.4.0 -> io.grpc:grpc-all:jar:1.2.0 -> io.grpc:grpc-core:jar:[1.2.0,1.2.0], org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.4.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.4.0 -> io.grpc:grpc-all:jar:1.2.0 -> io.grpc:grpc-okhttp:jar:1.2.0 -> io.grpc:grpc-core:jar:[1.2.0,1.2.0], org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.4.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.4.0 -> io.grpc:grpc-all:jar:1.2.0 -> io.grpc:grpc-protobuf-lite:jar:1.2.0 -> io.grpc:grpc-core:jar:1.2.0, org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.4.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.4.0 -> io.grpc:grpc-all:jar:1.2.0 -> io.grpc:grpc-protobuf-nano:jar:1.2.0 -> io.grpc:grpc-core:jar:1.2.0, org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.4.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.4.0 -> com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 -> io.grpc:grpc-core:jar:1.5.0, org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.4.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.4.0 -> com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 -> io.grpc:grpc-core:jar:1.7.0, org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.4.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.4.0 -> com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 -> io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 -> io.grpc:grpc-core:jar:1.6.1, com.google.cloud:google-cloud-language:jar:1.25.0 -> com.google.cloud:google-cloud-core-grpc:jar:1.25.0 -> io.grpc:grpc-protobuf:jar:1.10.1 -> io.grpc:grpc-core:jar:1.10.1, com.google.cloud:google-cloud-language:jar:1.25.0 -> io.grpc:grpc-netty-shaded:jar:1.10.1 -> io.grpc:grpc-core:jar:[1.10.1,1.10.1], com.google.cloud:google-cloud-language:jar:1.25.0 -> io.grpc:grpc-stub:jar:1.10.1 -> io.grpc:grpc-core:jar:1.10.1, com.google.cloud:google-cloud-language:jar:1.25.0 -> io.grpc:grpc-auth:jar:1.10.1 -> io.grpc:grpc-core:jar:[1.10.1,1.10.1]] -> [Help 1]

I would like to reproduce the issue, are you creating the pipeline from scratch or are you following any tutorial so I could start from there? — Xiaoxia Lin
i found that the problem is solved after i swap the dependency order for them. — John Watson

Phil Xu Phil Xu · Accepted Answer · 2018-07-08T07:29:34

I had exactly the same issue. You may solve it by using an earlier version of google-cloud-language, not ideal but works if you don't need the new features from natural language API.

    <dependency>
        <groupId>com.google.cloud</groupId>
        <artifactId>google-cloud-language</artifactId>
        <version>0.20.0-beta</version>
    </dependency>

0.20.0-beta is using io.grpc:grpc-core:jar:1.2.0. See if it works for you.

Unable to use dataflow with google cloud nature language API

1 Answers