Spark and cassandra, range query on clustering key

Question

I have cassandra table with following structure:

CREATE TABLE table ( key int, time timestamp, measure float, primary key (key, time) );

I need to create a Spark job which will read data from previous table, within specified start and end timestamp do some processing, and flush results back to cassandra.

So my spark-cassandra-connector will have to do a range query on clustering cassandra table column.

Are there any performance differences if I do:

sc.cassandraTable(keyspace,table).
as(caseClassObject).
filter(a => a.time.before(startTime) && a.time.after(endTime).....

so what I am doing is loading all the data into Spark and applying filtering

OR if I do this:

sc.cassandraTable(keyspace, table).
where(s"time>$startTime and time<$endTime)......

which filters all the data in Cassandra and then loads smaller subset to Spark.

The selectivity of a range query will be around 1% It is impossible to include partition key in the query.

Which of these two solutions is preferred?

RussS RussS · Accepted Answer · 2016-04-01T17:37:43

sc.cassandraTable(keyspace, table).where(s"time>$startTime and time<$endTime)

Will be MUCH faster. You are basically doing a percentage (if you only pull 5% of the data 5% of the total work) of the full grab in the first command to get the same data.

In the first case you are

Reading all of the data from Cassandra.
Serializing every object and then moving it to Spark.
Then finally filtering everything.

In the second case you are

Reading only the data you actually want from C*
Serializing only this tiny subset
There is no step 3

As an additional comment you can also put your case class type right in the call

sc.cassandraTable[CaseClassObject](keyspace, table)

Spark and cassandra, range query on clustering key

1 Answers