How we can deploy my existing kafka - spark - cassandra project to kafka - dataproc -cassandra in google-cloud-platform?

0
votes

My existing project is kafka-spark-cassandra. Now I have got gcp account and have to migrate spark jobs to dataproc. In my existing spark jobs parameters like masterip,memory,cores etc are passed through command line which is triggerd by a linux shell script and create new sparkConf.

val conf = new SparkConf(true) .setMaster(master) .setAppName("xxxx") .setJars(List(path+"/xxxx.jar")) .set("spark.executor.memory", memory) .set("spark.cores.max",cores) .set("spark.scheduler.mode", "FAIR") .set("spark.cassandra.connection.host", cassandra_ip)

1) How this can configure in dataproc?

2) Wheather there will be any compatibility issue b/w Spark 1.3(existing project) and Spark 1.6 provided by dataproc ? How it can resolve?

3) Is there any other connector needed for dataproc to get connected with Kafka and cassandra? I couldnt find any.

1
apache-kafkaspark-cassandra-connectorgoogle-cloud-dataproc

1 Answers

1
votes

1) When submitting a job, you can specify arguments and properties: https://cloud.google.com/sdk/gcloud/reference/dataproc/jobs/submit/spark. When determining which properties to set, keep in mind that Dataproc submits Spark jobs in yarn-client mode.

In general, this means you should avoid specifying master directly in code, instead letting it come from the spark.master value inside of spark-defaults.conf, and then your local setup would have that config set to local while Dataproc would automatically have it set to yarn-client with the necessary yarn config settings alongside it.

Likewise, keys like spark.executor.memory, etc., should make use of Spark's first-class command-line if running spark-submit directly:

spark-submit --conf spark.executor.memory=42G --conf spark.scheduler.mode=FAIR

or if submitting to Dataproc with gcloud:

gcloud dataproc jobs submit spark \
    --properties spark.executor.memory=42G,spark.scheduler.mode=FAIR

You'll also want to look at the equivalent --jars flags for jars instead of specifying it in code.

2) When building your project to deploy, ensure you exclude spark (e.g., in maven, mark spark as provided). You may hit compatibility issues, but without knowing all APIs in use, I can't say one way or the other. The simplest way to find out is to bump Spark to 1.6.1 in your build config and see what happens.

In general Spark core is considered GA and should thus be mostly backwards compatible in 1.X versions, but the compatibility guidelines didn't apply yet to subprojects like mllib and SparkSQL, so if you use those you're more likely to need to recompile against the newer Spark version.

3) Connectors should either be included in a fat jar, specified as --jars, or installed onto the cluster at creation via initialization actions.