Google Cloud Dataproc migration to Spark 1.6.0

0
votes

Will Google Dataproc start using Spark 1.6.0 anytime soon ? I'm creating a cluster using this command:

gcloud beta dataproc clusters create <cluster-name>

and it defaults to using Spark 1.5.2.

Thanks.

2
apache-sparkgoogle-cloud-dataproc

2 Answers

3
votes

You can expect to see a new image version for Google Cloud Dataproc which contains Spark 1.6.x within the coming weeks.

0
votes

https://cloud.google.com/dataproc/dataproc-versions lists the available versions. Currently you have a choice of Spark 1.5.0 or Spark 1.5.2 (the default).