Apache Zeppelin giving error with spark on different machines

Question

I have a local spark cluster setup with one master and one slave machine. I installed Zeppelin on more machine and trying to run some commands from Zeppelin to spark master machine. For that I have created a spark interpreter with master as spark://<ip>:7077

When I run sc command

It gives this error

java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.thrift.transport.TSocket.open(TSocket.java:182) at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51) at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37) at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60) at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861) at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435) at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:139) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:192) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:93) at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:207) at org.apache.zeppelin.scheduler.Job.run(Job.java:170) at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:304) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)

I can telnet port 7077 from zeppelin machine. Also this is my local vm machine so should not be issue with firewall, also disable all firewall.

Please let me know the issue.

Raveesh Sharma Raveesh Sharma · Accepted Answer · 2015-12-18T12:09:30

Perhaps you can go thorough the bug reported here https://issues.apache.org/jira/browse/ZEPPELIN-305 .

Taking an excerpt from that bug

I faced the same issue when I try to use Spark 1.5.1 under the latest version (0.6.0) of Zeppelin.

Following Dongjoon Hyun's suggestion, I set ZEPPELIN_MEM as -Xmx4g (the same value as SPARK_DRIVER_MEMORY in spark/conf/spark-env.sh) in zeppelin/conf/zeppelin-env.sh

export ZEPPELIN_MEM=-Xmx4g

As a result, the java.net.ConnectException problem is solved. Thanks!

Apache Zeppelin giving error with spark on different machines

1 Answers