installing cassandra-spark-connector - but getting error creating SparkContext

0
votes

installing cassandra-spark-connector - but getting error creating SparkContext

Please help. I am following the guide - https://github.com/datastax/spark-cassandra-connector/blob/master/doc/0_quick_start.md

Env - Spark 1.0.1, Scala 2.10.4

But having the following error message when i get to creating SparkContext. The last line says all master are unresponsive, giving up. Master is still running

My steps are: ./sbin/start-all - starts all workes successfully

MASTER=spark://spark-master-hostname:7077 ./bin/spark-shell - to lunch spark on the master



scala> import org.apache.spark.SparkContext
import org.apache.spark.SparkContext

scala> import org.apache.spark.SparkContext._
import org.apache.spark.SparkContext._

scala> import org.apache.spark.SparkConf
import org.apache.spark.SparkConf

scala> val conf = new SparkConf(true).set("spark.cassandra.connection.host","cassandra-host-ip")
conf: org.apache.spark.SparkConf = org.apache.spark.SparkConf@9f073

*scala> val sc = new SparkContext("spark://spark-master-ipaddress:7077", "test", conf)*
**14/07/29 12:18:23 WARN AbstractLifeCycle: FAILED
[email protected]:4040: java.net.BindException: Address already in use
java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:444)
    at sun.nio.ch.Net.bind(Net.java:436)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
    at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
   at   org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.eclipse.jetty.server.Server.doStart(Server.java:293)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply$mcV$sp(JettyUtils.scala:192)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:192)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:192)
    at scala.util.Try$.apply(Try.scala:161)
    at org.apache.spark.ui.JettyUtils$.connect$1(JettyUtils.scala:191)
    at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:205)
    at org.apache.spark.ui.WebUI.bind(WebUI.scala:99)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:223)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:97)
    at $line15.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:17)
    at $line15.$read$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:22)
    at $line15.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:24)
    at $line15.$read$$iwC$$iwC$$iwC.<init>(<console>:26)
    at $line15.$read$$iwC$$iwC.<init>(<console>:28)
    at $line15.$read$$iwC.<init>(<console>:30)
    at $line15.$read.<init>(<console>:32)
    at $line15.$read$.<init>(<console>:36)
    at $line15.$read$.<clinit>(<console>)
    at $line15.$eval$.<init>(<console>:7)
    at $line15.$eval$.<clinit>(<console>)
    at $line15.$eval.$print(<console>)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:788)
    at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1056)
    at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:614)
    at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:645)
    at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:609)
    at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:796)
    at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:841)
    at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:753)
    at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:601)
    at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:608)
    at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:611)
    at   org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:936)
    at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
    at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
    at     scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
    at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:884)
    at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:982)
    at org.apache.spark.repl.Main$.main(Main.scala:31)
    at org.apache.spark.repl.Main.main(Main.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at  sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
    14/07/29 12:18:23 WARN AbstractLifeCycle: FAILED     org.eclipse.jetty.server.Server@dd53c8a:     java.net.BindException: Address already in use
    java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:444)
    at sun.nio.ch.Net.bind(Net.java:436)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at  org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
    at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
    at  org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265        )
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.eclipse.jetty.server.Server.doStart(Server.java:293)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply$mcV$sp(JettyUtils.scala:192)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:192)
    at org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:192)
    at scala.util.Try$.apply(Try.scala:161)
    at org.apache.spark.ui.JettyUtils$.connect$1(JettyUtils.scala:191)
    at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:205)
    at org.apache.spark.ui.WebUI.bind(WebUI.scala:99)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:223)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:97)
    at $line15.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:17)
    at $line15.$read$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:22)
    at $line15.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:24)
    at $line15.$read$$iwC$$iwC$$iwC.<init>(<console>:26)
    at $line15.$read$$iwC$$iwC.<init>(<console>:28)
    at $line15.$read$$iwC.<init>(<console>:30)
    at $line15.$read.<init>(<console>:32)
    at $line15.$read$.<init>(<console>:36)
    at $line15.$read$.<clinit>(<console>)
    at $line15.$eval$.<init>(<console>:7)
    at $line15.$eval$.<clinit>(<console>)
    at $line15.$eval.$print(<console>)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at   sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:788)
    at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1056)
    at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:614)
    at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:645)
    at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:609)
    at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:796)
    at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:841)
    at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:753)
    at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:601)
    at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:608)
    at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:611)
    at     org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:936)
    at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
    at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
    at     scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
    at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:884)
    at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:982)
    at org.apache.spark.repl.Main$.main(Main.scala:31)
    at org.apache.spark.repl.Main.main(Main.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
    sc: org.apache.spark.SparkContext = org.apache.spark.SparkContext@4353d65f
    scala> 14/07/29 12:19:24 ERRstrong textOR SparkDeploySchedulerBackend: Application     has been killed. Reason: All master**s are unresponsive! Giving up.
    14/07/29 12:19:24 ERROR TaskSchedulerImpl: Exiting due to error from cluster     scheduler: All masters are unresponsive! Giving up.****
2
apache-sparkscala-2.10
you need to replace placeholders like spark-master-ipaddress and spark-master-hostname in the example with actual values – maasg
Thank you. That is correct. Just what i did. Replaced sparkmaster with hostname ans also with ipaddress, non worked. – del

2 Answers

0
votes

Step 1 : To load the connector into the Spark Shell, start the shell with this command: 

../bin/spark-shell –jars ~/apps/spark-1.2/jars/spark-cassandra-connector-assembly-1.1.1-SNAPSHOT.jar

Step 2 : Connect the Spark Context to the Cassandra cluster.Stop the default context.

sc.stop

Step 3 :Import the necessary jar files.

import com.datastax.spark.connector._, org.apache.spark.SparkContext, org.apache.spark.SparkContext._, org.apache.spark.SparkConf

Step 4 : Make a new SparkConf with the Cassandra connection details:

val conf = new SparkConf(true).set("spark.cassandra.connection.host", "localhost")

Step 5 : Create a new Spark Context:

  val sc = new SparkContext(conf)

You now have a new SparkContext which is connected to your Cassandra cluster.

0
votes

Have tried to use spark-packages?

Spark Cassandra Connector on spark-packages.org

Boils down to

$SPARK_HOME/bin/spark-shell --packages datastax:spark-cassandra-connector:2.0.0-M2-s_2.10

where you need to use the correct version for your version of spark. This should setup everything needed automatically.