Cassandra connection to Spark

0
votes

I am connecting spark with Cassandra and I am storing csv file in Cassandra, when I enter this command I got error.

 dfprev.write.format("org.apache.spark.sql.cassandra") .options(Map("keyspace"->"sensorkeyspace","table"->"sensortable")).save()

Then I got this error.

java.io.IOException: Failed to open native connection to Cassandra at {127.0.0.1}:9042 at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:168) at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$8.apply(CassandraConnector.scala:154) at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$8.apply(CassandraConnector.scala:154) at com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:32) at com.datastax.spark.connector.cql.RefCountedCache.syncAcquire(RefCountedCache.scala:69)

3
scalaapache-sparkcassandraapache-spark-sqlspark-cassandra-connector
Is 127.0.0.1 and port 9042 your cassandra connection? – gasparms
I am using Cassandra 3.11.2 and spark 2.0.2 and scala version 2.11.8, would you tell me which connector I used. – Real tiger
Did you ever get this working? I'm facing the same issue. – Gorgon_Union

3 Answers

0
votes

Are you Cassandra listening on localhost? You may need to configure the list of IP addresses of your Cassandra cluster by specifying spark.cassandra.connection.host setting in Spark configuration. See documentation for details.

0
votes

There might any one of the following

  1. Cassandra server might not be running at 127.0.0.1:9042

    Please check the cassandra is listening at port 9042 using netstat -an command.

  2. There might be dependency issues when fat jar.

    Please make sure that you have added right version of cassandra connector in library dependency like

    "com.datastax.spark" %% "spark-cassandra-connector" % "2.0.0-M3"

I am running this command ./spark-shell --packages com.datastax.spark:spark-cassandra-connector_2.11:2.0.0-M3 --conf spark.cassandra.connection.host=127.0.0.

The package to be specified as,

spark-shell --packages "com.datastax.spark":"spark-cassandra-connector_2.11":"2.0.0-M3"

0
votes

Check these things, May solve your problem, 1. Find cqlsh.py file in your system by entering the below command in shell

 whereis cqlsh

  1. Edit the cqlsh.py and change the DEFAULT PORT to your IP

  2. Initiate spark context with the following SparkConfig()

      val conf = new SparkConf().set("spark.cassandra.connection.host", "<YOUR IP>")
  val sc = new SparkContext(conf)