I have an intermittent issue using EMR (Release label: emr-5.8.0) on Amazon Web Services. I run the following services:
- Hadoop
- Yarn
- Hive
- Sqoop
- Spark
- Hive on Spark
The error is the following
Connection refused: ip-10-128-196-59/10.128.196.59:42560
The full stack trace is:
18:28:59.495 [Driver] ERROR org.apache.spark.deploy.yarn.ApplicationMaster - User class threw exception: java.util.concurrent.ExecutionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: ip-10-128-196-59/10.128.196.59:42560 java.util.concurrent.ExecutionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: ip-10-128-196-59/10.128.196.59:42560 at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:41) ~[netty-all-4.0.43.Final.jar:4.0.43.Final] at org.apache.hive.spark.client.RemoteDriver.(RemoteDriver.java:145) ~[hive-exec-2.3.0-amzn-0.jar:2.3.0-amzn-0] at org.apache.hive.spark.client.RemoteDriver.main(RemoteDriver.java:516) ~[hive-exec-2.3.0-amzn-0.jar:2.3.0-amzn-0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_141] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_141] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_141] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141] at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:635) [spark-yarn_2.11-2.2.0.jar:2.2.0] Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: ip-10-128-196-59/10.128.196.59:42560 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_141] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:1.8.0_141] at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:257) ~[netty-all-4.0.43.Final.jar:4.0.43.Final] at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:291) ~[netty-all-4.0.43.Final.jar:4.0.43.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:631) ~[netty-all-4.0.43.Final.jar:4.0.43.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:566) ~[netty-all-4.0.43.Final.jar:4.0.43.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:480) ~[netty-all-4.0.43.Final.jar:4.0.43.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442) ~[netty-all-4.0.43.Final.jar:4.0.43.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) ~[netty-all-4.0.43.Final.jar:4.0.43.Final] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_141] 18:28:59.511 [main] ERROR org.apache.spark.deploy.yarn.ApplicationMaster - Uncaught exception: org.apache.spark.SparkException: Exception thrown in awaitResult: at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:205) ~[spark-core_2.11-2.2.0.jar:2.2.0] at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:401) [spark-yarn_2.11-2.2.0.jar:2.2.0] at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:254) [spark-yarn_2.11-2.2.0.jar:2.2.0] at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:764) [spark-yarn_2.11-2.2.0.jar:2.2.0] at org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:67) [spark-core_2.11-2.2.0.jar:2.2.0] at org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:66) [spark-core_2.11-2.2.0.jar:2.2.0] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_141] at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_141] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) [hadoop-common-2.7.3-amzn-3.jar:?] at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:66) [spark-core_2.11-2.2.0.jar:2.2.0] at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:762) [spark-yarn_2.11-2.2.0.jar:2.2.0] at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala) [spark-yarn_2.11-2.2.0.jar:2.2.0] Caused by: java.util.concurrent.ExecutionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: ip-10-128-196-59/10.128.196.59:42560 at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:41) ~[netty-all-4.0.43.Final.jar:4.0.43.Final] at org.apache.hive.spark.client.RemoteDriver.(RemoteDriver.java:145) ~[hive-exec-2.3.0-amzn-0.jar:2.3.0-amzn-0] at org.apache.hive.spark.client.RemoteDriver.main(RemoteDriver.java:516) ~[hive-exec-2.3.0-amzn-0.jar:2.3.0-amzn-0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_141] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_141] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_141] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141] at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:635) ~[spark-yarn_2.11-2.2.0.jar:2.2.0] Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: ip-10-128-196-59/10.128.196.59:42560 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_141] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:1.8.0_141] at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:257) ~[netty-all-4.0.43.Final.jar:4.0.43.Final] at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:291) ~[netty-all-4.0.43.Final.jar:4.0.43.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:631) ~[netty-all-4.0.43.Final.jar:4.0.43.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:566) ~[netty-all-4.0.43.Final.jar:4.0.43.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:480) ~[netty-all-4.0.43.Final.jar:4.0.43.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442) ~[netty-all-4.0.43.Final.jar:4.0.43.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) ~[netty-all-4.0.43.Final.jar:4.0.43.Final] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_141]
There's nothing running on port 42560 on the machine with IP 10.128.196.59. I'm using all the default settings from AWS, and unfortunately I can't find AWS documentation regarding port 42560
[EDIT] If you run the process again, the port is different?!
