Data loss Spark 2.1 -kafka broker 0.8.2.1 streaming

0
votes

1 streaming and Kafka broker version 0.8.2.1, I have separate servers for spark and kafka on AWS.

Using val directKafkaStream = KafkaUtils.createDirectStream direct approach. StreamingContext(conf, Seconds(300)), I am expecting to get 30 string from streaming but actual receiving only 15-25 in range . Cross check kafka consumer on same topic showing 30 string during 300 seconds. And stream.foreachRDD { rdd => giving 15to 20 strings.

What is wrong behind getting uneventual data. I am using sparksession creating sc and ssc.

Thank You.

1
apache-sparkapache-kafkaspark-streaming

1 Answers

0
votes

add auto.offset.reset to smallest in kafka param

 val kafkaParams = Map[String, String](
         "auto.offset.reset" -> "smallest", ......)