WriteConf of Spark-Cassandra Connector being used or not

Question

I am using Spark version 1.6.2, Spark-Cassandra Connector 1.6.0, Cassandra-Driver-Core 3.0.3

I am writing a simple Spark job in which I am trying to insert some rows to a table in Cassandra. The code snippet used was:

val sparkConf = (new SparkConf(true).set("spark.cassandra.connection.host", "<Cassandra IP>")
.set("spark.cassandra.auth.username", "test")
.set("spark.cassandra.auth.password", "test")
.set("spark.cassandra.output.batch.size.rows", "1"))

val sc = new SparkContext(sparkConf)
val cassandraSQLContext = new CassandraSQLContext(sc)
cassandraSQLContext.setKeyspace("test")
val query = "select * from test"
val dataRDD = cassandraSQLContext.cassandraSql(query).rdd

val addRowList = (ListBuffer(
Test(111, 10, 100000, "{'test':'0','test1':'1','others':'2'}"),
Test(111, 20, 200000, "{'test':'0','test1':'1','others':'2'}")
))

val insertRowRDD = sc.parallelize(addRowList)
insertRowRDD.saveToCassandra("test", "test")

Test() is a case class

Now, I have passed the WriteConf parameter output.batch.size.rows when making sparkConf object. I am expecting that this code will write 1 row in a batch at a time in Cassandra. I am not getting any method through which I can cross verify that the configuration of writing a batch in cassandra is not the default one but the one passed in the code snippet.

I could not find anything in the cassandra cassandra.log, system.log and debug.log

So can anyone help me with the method of cross verifying the WriteConf being used by Spark-Cassandra Connector to write batches in Cassandra?

RussS RussS · Accepted Answer · 2017-01-24T19:03:47

There are two things you can do to verify that your setting was correctly set.

First you can call the method which creates WriteConf
```
WriteConf.fromSparkConf(sparkConf)
```
The resulting object can be inspected to make sure all the values are what you want. This is the default arg to SaveToCassandra

You can explicitly pass a WriteConf to the saveToCassandraMethod

saveAsCassandraTable(keyspace, table, writeConf = WriteConf(...))

WriteConf of Spark-Cassandra Connector being used or not

1 Answers