Running complex SQL queries on Cassandra tables using Spark SQL

1
votes

hereI have setup Cassandra and Spark with cassandra- spark connector. I am able to create RDDs using Scala. But I would like to run complex SQL queries (Aggregation/Analytical functions/Window functions) using Spark SQL on Cassandra tables , could you help how should I proceed ?getting error like this

following is the query used : sqlContext.sql( """CREATE TEMPORARY TABLE words |USING org.apache.spark.sql.cassandra |OPTIONS ( | table "words", | keyspace "test", | cluster "Test Cluster", | pushdown "true" |)""".stripMargin)

below is the error :[enter image description here][2]

new error: enter image description here

2
scalaapache-sparkcassandraapache-spark-sql
Spark SQL is rich compared to CQL , so would like to implement analytics using this. - Goutham
I am able to get the rdds created for Cassandra tables using scala. (I am using Ubuntu terminal), but couldn't get the exact procedure to query Cassandra tables using spark-sql. could you help - Goutham
following command used to start spark -sql based on one of the questions : stackoverflow.com/questions/30293070/… but getting error . $SPARK_HOME/bin/spark-sql --packages datastax:spark-cassandra-connector:2.0.0-s_2.11 --conf spark.cassandra.connection.host=127.0.0.1 - Goutham
hi included the error screenshot and the query used , please check - Goutham

2 Answers

0
votes

First thing I noticed from your post is that , sqlContext.sql(...) used in your query but your screenshot shows sc.sql(...).

I take screenshot content as your actual issue. In Spark shell, Once you've loaded the shell, both the SparkContext (sc) and the SQLContext (sqlContext) are already loaded and ready to go. sql(...) does't exit in SparkContext so you should try with sqlContext.sql(...).

0
votes

Most probably in your spark-shell context started as Spark Session and value for that is spark. Try your commands with spark instead of sqlContext.