Error running spark app using spark-cassandra connector

Question

I have written a basic spark app that reads and writes to Cassandra following this guide (https://github.com/datastax/spark-cassandra-connector/blob/master/doc/0_quick_start.md)

This is what the .sbt for this app looks like:

name := "test Project"

version := "1.0"

scalaVersion := "2.10.5"

libraryDependencies ++= Seq(
      "org.apache.spark" %% "spark-core" % "1.2.1",
      "com.google.guava" % "guava" % "14.0.1",
      "com.datastax.spark" %% "spark-cassandra-connector" % "1.2.1",
      "org.apache.cassandra" % "cassandra-thrift" % "2.0.14",
      "org.apache.cassandra" % "cassandra-clientutil" % "2.0.14",
      "com.datastax.cassandra" % "cassandra-driver-core"  % "2.0.14"
)

As you can see the Spark version is 1.2.1 (and not 1.3.1 like a lot of other questions) but when I run this app using spark-submit I still run into the error:

WARN TaskSetManager: Lost task 1.0 in stage 0.0 (TID 1, abcdev26): java.lang.NoSuchMethodError: org.apache.spark.executor.TaskMetrics.inputMetrics_$eq(Lscala/Option;)V
        at com.datastax.spark.connector.metrics.InputMetricsUpdater$.apply(InputMetricsUpdater.scala:61)
        at com.datastax.spark.connector.rdd.CassandraTableScanRDD.compute(CassandraTableScanRDD.scala:196)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
        at org.apache.spark.scheduler.Task.run(Task.scala:64)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

What am I missing? All the answered I've searched up so far suggest using 1.2.1 which I already am doing.

Any suggestions would be much appreciated!

How is your cluster setup? Are you 100% sure that you are running against Spark 1.2.1? Also on the executors? The problem is that this metric accessor became private in Spark 1.3.0 and therefore cannot be found at runtime. See [github.com/apache/spark/blob/v1.2.2/core/src/main/scala/org/… v1.2.2) vs [github.com/apache/spark/blob/v1.3.0/core/src/main/scala/org/… v1.3.0) — maasg
Sorry doing some more looking around I realize I am running spark 1.3.1. In order to go back to spark 1.2.x would I have to download that version and start from scratch? Or can I just change the revision values in the .sbt file? — user3376961
I can't know what you did before. You will need to setup a cluster environment for 1.2.x probably following the same process you did before. — maasg

maasg maasg · Accepted Answer · 2015-06-09T20:08:13

Are you 100% sure that you are running against Spark 1.2.1? Also on the executors?

The problem is that this metric accessor became private in Spark 1.3.0 and therefore cannot be found at runtime. See TaskMetrics.scala - Spark 1.2.2 vs TaskMetrics.scala - spark v1.3.0, so most probably there's a Spark1.3.x version somewhere.

Make sure the same version 1.2.x is on all executors as well.

Error running spark app using spark-cassandra connector

1 Answers