Error in Dataflow: io.grpc.StatusRuntimeException: UNAVAILABLE

Question

I'm trying to build a Dataflow job that reads from BigQuery and sends messages to Pubsub topics. I've been struggling with inter-dependencies, and this is the latest issue.

The Dataflow job starts fine, but fails with the following exception:

com.google.cloud.pubsub.PubSubException: io.grpc.StatusRuntimeException: UNAVAILABLE at com.google.cloud.pubsub.spi.DefaultPubSubRpc$1.apply(DefaultPubSubRpc.java:174) at com.google.cloud.pubsub.spi.DefaultPubSubRpc$1.apply(DefaultPubSubRpc.java:168) at com.google.common.util.concurrent.Futures$CatchingFuture.doFallback(Futures.java:842) at com.google.common.util.concurrent.Futures$CatchingFuture.doFallback(Futures.java:834) at com.google.common.util.concurrent.Futures$AbstractCatchingFuture.run(Futures.java:789) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:634) at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:53) at com.google.api.gax.grpc.RetryingCallable$Retryer$1.onFailure(RetryingCallable.java:139) at com.google.common.util.concurrent.Futures$6.run(Futures.java:1764) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:634) at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:53) at com.google.api.gax.grpc.ExceptionTransformingCallable$1.onFailure(ExceptionTransformingCallable.java:91) at com.google.common.util.concurrent.Futures$6.run(Futures.java:1764) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:634) at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:382) at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:358) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$3.runInContext(ClientCallImpl.java:462) at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:54) at io.grpc.internal.SerializingExecutor$TaskRunner.run(SerializingExecutor.java:154) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) Caused by: com.google.api.gax.grpc.ApiException: io.grpc.StatusRuntimeException: UNAVAILABLE at com.google.api.gax.grpc.ExceptionTransformingCallable$1.onFailure(ExceptionTransformingCallable.java:91) at com.google.common.util.concurrent.Futures$6.run(Futures.java:1764) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:634) at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:382) at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:358) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$3.runInContext(ClientCallImpl.java:462) at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:54) at io.grpc.internal.SerializingExecutor$TaskRunner.run(SerializingExecutor.java:154) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE at io.grpc.Status.asRuntimeException(Status.java:431) ... 11 more Caused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate) at sun.security.ssl.Handshaker.activate(Handshaker.java:503) at sun.security.ssl.SSLEngineImpl.kickstartHandshake(SSLEngineImpl.java:729) at sun.security.ssl.SSLEngineImpl.beginHandshake(SSLEngineImpl.java:756) at io.netty.handler.ssl.JdkSslEngine.beginHandshake(JdkSslEngine.java:147) at io.netty.handler.ssl.SslHandler.handshake(SslHandler.java:1363) at io.netty.handler.ssl.SslHandler.channelActive(SslHandler.java:1405) at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelActiveNow(ChannelHandlerInvokerUtil.java:48) at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelActive(DefaultChannelHandlerInvoker.java:79) at io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:126) at io.netty.channel.DefaultChannelPipeline.fireChannelActive(DefaultChannelPipeline.java:884) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:260) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:290) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:527) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:467) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:381) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:742) ... 3 more

I have a suspicion that it's related to the netty/ssl libraries being used, but I have no idea what the right course of action is. I've spent a lot of time trying to exclude certain versions of certain libraries from the resulting shaded jar where classes overlapped, but ended up with class not found exceptions instead.

These are my dependencies:

<dependencies>
  <dependency>
    <groupId>com.google.cloud.dataflow</groupId>
    <artifactId>google-cloud-dataflow-java-sdk-all</artifactId>
    <version>1.8.0</version>
  </dependency>
  <dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>google-cloud-pubsub</artifactId>
    <version>0.3.0</version>
  </dependency>
  <dependency>
    <groupId>com.google.protobuf</groupId>
    <artifactId>protobuf-java</artifactId>
    <version>3.0.0-beta-1</version>
  </dependency>
</dependencies>

And this is the dependency tree:

[INFO] --- maven-dependency-plugin:2.1:tree (default-cli) @ dataflow-bulk-load ---
[INFO] com.google.lindsaysmith.titan:dataflow-bulk-load:jar:0.0.1-SNAPSHOT
[INFO] +- com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:1.8.0:compile
[INFO] |  +- com.google.apis:google-api-services-dataflow:jar:v1b3-rev36-1.22.0:compile
[INFO] |  +- io.grpc:grpc-all:jar:0.13.1:compile
[INFO] |  |  +- io.grpc:grpc-auth:jar:0.13.1:compile
[INFO] |  |  +- io.grpc:grpc-netty:jar:0.13.1:compile
[INFO] |  |  |  \- io.netty:netty-codec-http2:jar:4.1.0.CR1:compile
[INFO] |  |  |     \- io.netty:netty-codec-http:jar:4.1.0.CR1:compile
[INFO] |  |  +- io.grpc:grpc-protobuf:jar:0.13.1:compile
[INFO] |  |  +- io.grpc:grpc-core:jar:0.13.1:compile
[INFO] |  |  +- io.grpc:grpc-okhttp:jar:0.13.1:compile
[INFO] |  |  |  +- com.squareup.okio:okio:jar:1.6.0:compile
[INFO] |  |  |  \- com.squareup.okhttp:okhttp:jar:2.5.0:compile
[INFO] |  |  +- io.grpc:grpc-protobuf-nano:jar:0.13.1:compile
[INFO] |  |  |  \- com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5:compile
[INFO] |  |  \- io.grpc:grpc-stub:jar:0.13.1:compile
[INFO] |  +- io.netty:netty-handler:jar:4.1.0.CR1:compile
[INFO] |  |  +- io.netty:netty-buffer:jar:4.1.0.CR1:compile
[INFO] |  |  |  \- io.netty:netty-common:jar:4.1.0.CR1:compile
[INFO] |  |  +- io.netty:netty-transport:jar:4.1.0.CR1:compile
[INFO] |  |  |  \- io.netty:netty-resolver:jar:4.1.0.CR1:compile
[INFO] |  |  \- io.netty:netty-codec:jar:4.1.0.CR1:compile
[INFO] |  +- com.google.api.grpc:grpc-pubsub-v1:jar:0.0.2:compile
[INFO] |  |  \- com.google.api.grpc:grpc-core-proto:jar:0.0.3:compile
[INFO] |  +- com.google.auth:google-auth-library-oauth2-http:jar:0.4.0:compile
[INFO] |  |  \- com.google.auth:google-auth-library-credentials:jar:0.4.0:compile
[INFO] |  +- com.google.cloud.bigtable:bigtable-protos:jar:0.3.0:compile
[INFO] |  |  \- com.google.guava:guava:jar:19.0:compile
[INFO] |  +- com.google.api-client:google-api-client:jar:1.22.0:compile
[INFO] |  +- com.google.apis:google-api-services-bigquery:jar:v2-rev295-1.22.0:compile
[INFO] |  +- com.google.apis:google-api-services-clouddebugger:jar:v2-rev8-1.22.0:compile
[INFO] |  +- com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0:compile
[INFO] |  +- com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0:compile
[INFO] |  +- com.google.http-client:google-http-client:jar:1.22.0:compile
[INFO] |  |  \- org.apache.httpcomponents:httpclient:jar:4.0.1:compile
[INFO] |  |     +- org.apache.httpcomponents:httpcore:jar:4.0.1:compile
[INFO] |  |     +- commons-logging:commons-logging:jar:1.1.1:compile
[INFO] |  |     \- commons-codec:commons-codec:jar:1.3:compile
[INFO] |  +- com.google.http-client:google-http-client-jackson2:jar:1.22.0:compile
[INFO] |  +- com.google.oauth-client:google-oauth-client-java6:jar:1.22.0:compile
[INFO] |  +- com.google.oauth-client:google-oauth-client:jar:1.22.0:compile
[INFO] |  +- com.google.apis:google-api-services-datastore-protobuf:jar:v1beta2-rev1-4.0.0:compile
[INFO] |  +- com.google.cloud.bigdataoss:gcsio:jar:1.4.5:compile
[INFO] |  |  +- com.google.api-client:google-api-client-java6:jar:1.22.0:compile
[INFO] |  |  \- com.google.api-client:google-api-client-jackson2:jar:1.22.0:compile
[INFO] |  +- com.google.cloud.datastore:datastore-v1-proto-client:jar:1.1.0:compile
[INFO] |  |  +- com.google.http-client:google-http-client-protobuf:jar:1.22.0:compile
[INFO] |  |  \- com.google.http-client:google-http-client-jackson:jar:1.22.0:compile
[INFO] |  +- com.google.cloud.datastore:datastore-v1-protos:jar:1.0.1:compile
[INFO] |  +- com.google.cloud.bigdataoss:util:jar:1.4.5:compile
[INFO] |  +- com.google.code.findbugs:jsr305:jar:3.0.1:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-core:jar:2.7.0:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.7.0:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.7.0:compile
[INFO] |  +- org.slf4j:slf4j-api:jar:1.7.14:compile
[INFO] |  +- org.apache.avro:avro:jar:1.7.7:compile
[INFO] |  |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] |  |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] |  |  \- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
[INFO] |  +- org.xerial.snappy:snappy-java:jar:1.1.2.1:compile
[INFO] |  +- org.apache.commons:commons-compress:jar:1.9:compile
[INFO] |  \- joda-time:joda-time:jar:2.4:compile
[INFO] +- com.google.cloud:google-cloud-pubsub:jar:0.3.0:compile
[INFO] |  +- io.netty:netty-tcnative-boringssl-static:jar:1.1.33.Fork17:compile
[INFO] |  +- com.google.cloud:google-cloud-core:jar:0.3.0:compile
[INFO] |  |  +- com.google.api-client:google-api-client-appengine:jar:1.21.0:compile
[INFO] |  |  |  +- com.google.oauth-client:google-oauth-client-appengine:jar:1.21.0:compile
[INFO] |  |  |  |  +- com.google.oauth-client:google-oauth-client-servlet:jar:1.21.0:compile
[INFO] |  |  |  |  |  \- com.google.http-client:google-http-client-jdo:jar:1.21.0:compile
[INFO] |  |  |  |  \- javax.servlet:servlet-api:jar:2.5:compile
[INFO] |  |  |  +- com.google.api-client:google-api-client-servlet:jar:1.21.0:compile
[INFO] |  |  |  |  \- javax.jdo:jdo2-api:jar:2.3-eb:compile
[INFO] |  |  |  |     \- javax.transaction:transaction-api:jar:1.1:compile
[INFO] |  |  |  \- com.google.http-client:google-http-client-appengine:jar:1.21.0:compile
[INFO] |  |  +- org.json:json:jar:20151123:compile
[INFO] |  |  +- com.google.api:gax:jar:0.0.16:compile
[INFO] |  |  |  \- com.google.inject:guice:jar:4.0:compile
[INFO] |  |  |     +- javax.inject:javax.inject:jar:1:compile
[INFO] |  |  |     \- aopalliance:aopalliance:jar:1.0:compile
[INFO] |  |  \- com.google.api.grpc:grpc-google-common-protos:jar:0.0.7:compile
[INFO] |  +- com.google.api.grpc:grpc-google-pubsub-v1:jar:0.0.7:compile
[INFO] |  \- com.google.auto.value:auto-value:jar:1.1:compile
[INFO] \- com.google.protobuf:protobuf-java:jar:3.0.0-beta-1:compile
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.749s
[INFO] Finished at: Sun Dec 04 16:50:41 CET 2016
[INFO] Final Memory: 16M/972M
[INFO] ------------------------------------------------------------------------

If this can be reliably reproduced, you might consider posting a copy of the app's code (stripped down to the most minimal level possible while still reproducing, being sure to scrub anything proprietary or sensitive) to the Dataflow github Issues Page, so Google can take a look at the cause. — Nick
I recreated a whole new project, and this issue never occurred again. I think this was related to some automated firewall settings that were applied to my project, possibly breaking or disrupting dns requests. — Lindsay Smith
Alright, that's at least good to hear. But in that case, I'm not quite sure if that qualifies for a self-answer... I suppose given that it's unlikely to recur, but still useful for others to find, it wouldn't hurt to either leave it without an answer or post a self-answer, either way. — Nick
Actually I just realized this was a different issue :/. But I did solve this, I'll post the self answer. — Lindsay Smith

Lindsay Smith Lindsay Smith · Accepted Answer · 2017-01-18T13:48:32

This issue was occurring when I was trying to use the PubSub libraries to manually send messages for each processed item in DataFlow.

Instead of doing that, I switched to using DataFlow's PubSub IO Sink class to send the messages, and this worked flawlessly.

Docs:

https://cloud.google.com/dataflow/model/pubsub-io#writing-with-pubsubio

Error in Dataflow: io.grpc.StatusRuntimeException: UNAVAILABLE

1 Answers