How to run Kafka as a stream for Apache Spark using Scala 2.11?

1
votes

I haven't been able to find any any build of Spark Streaming integration for Kafka for Scala 2.11. There is a one available here http://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka_2.10 but it is for 2.10

Can anyone point me to a 2.11 build?

1
scalaapache-sparkapache-kafka

1 Answers

2
votes

Specific to your question:

It's not feasible to run Spark Kafka against Scala 2.11 by now (Spark-1.3)

General method to build from source code:

If no pre-build version available, you could build spark yourself and fulfil your needs by specifying some build parameters.

The detailed build procedure could be find: Building Spark

In short, It would only take two steps if build against scala-2.10:

export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"

mvn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package

You should specify profiles or properties that fits your situation in the second command

Note the part Building Spark states on Building for Scala 2.11:

To produce a Spark package compiled with Scala 2.11, use the -Dscala-2.11 property:

dev/change-version-to-2.11.sh
mvn -Pyarn -Phadoop-2.4 -Dscala-2.11 -DskipTests clean package

Scala 2.11 support in Spark does not support a few features due to dependencies which are themselves not Scala 2.11 ready. Specifically, Spark’s external Kafka library and JDBC component are not yet supported in Scala 2.11 builds.