Why does my Spark Streaming application not print the number of records from Kafka (using count operator)?

Question

I am working on a spark application which needs to read data from Kafka. I created a Kafka topic where producer was posting messages. I verified from console consumer that messages were successfully posted .

I wrote a short spark application to read data from Kafka, but it is not getting any data. Following is the code i used:

def main(args: Array[String]): Unit = {
   val Array(zkQuorum, group, topics, numThreads) = args
   val sparkConf = new SparkConf().setAppName("SparkConsumer").setMaster("local[2]")
   val ssc = new StreamingContext(sparkConf, Seconds(2))

   val topicMap = topics.split(",").map((_, numThreads.toInt)).toMap
   val lines = KafkaUtils.createStream(ssc, zkQuorum, group, topicMap).map(_._2)

   process(lines) // prints the number of records in Kafka topic

   ssc.start()
   ssc.awaitTermination()
 }

 private def process(lines: DStream[String]) { 
   val z = lines.count()
   println("count of lines is "+z) 
    //edit
   lines.foreachRDD(rdd => rdd.map(println) 
   // <-- Why does this **not** print?
 )

Any suggestions on how to resolve this issue?

******EDIT****

I have used

lines.foreachRDD(rdd => rdd.map(println)

as well in actual code but that is also not working. I set the retention period as mentioned in post : Kafka spark directStream can not get data . But still the problem exist.

Jacek Laskowski Jacek Laskowski · Accepted Answer · 2016-11-29T10:21:35

Your process is a continuation of a DStream pipeline with no output operator that gets the pipeline executed every batch interval.

You can "see" it by reading the signature of count operator:

count(): DStream[Long]

Quoting the count's scaladoc:

Returns a new DStream in which each RDD has a single element generated by counting each RDD of this DStream.

So, you have a dstream of Kafka records that you transform to a dstream of single values (being the result of count). Not much to have it outputed (to a console or any other sink).

You have to end the pipeline using an output operator as described in the official documentation Output Operations on DStreams:

Output operations allow DStream’s data to be pushed out to external systems like a database or a file systems. Since the output operations actually allow the transformed data to be consumed by external systems, they trigger the actual execution of all the DStream transformations (similar to actions for RDDs).

(Low-Level) Output operators register input dstreams as output dstreams so the execution can start. Spark Streaming's DStream by design has no notion of being an output dstream. It is DStreamGraph to know and be able to differentiate between input and output dstreams.

Why does my Spark Streaming application not print the number of records from Kafka (using count operator)?

1 Answers