Spark streaming : batch processing time slowly increase

2
votes

I use spark with the cassandra spark connector and direct kafka.

And I seed batch procession increasing slowly over the time. Even when there is nothing to process incoming from kafka.

I think it is about a few milliseconds by batch, but after a long time, a batch can take several more seconds until it reaches the batch interval and finally crash.

I thought first it was a memory leak, but I think the processing time would be less linear but exponentially instead.

I don't really know if it is stages that become longer and longer or the latency between stage that increases.

I use spark 1.4.0

exemple

Any pointers about this?

EDIT : A attentive look at the evolution of the processing time of each batch, comparing total jobs processing time.

And it appears that even if batch processing time increases, the job processing time are not increasing. exemple : for a batch that take 7s the sum of each job processing time is 1.5s. (as shown in the image below)

Is it because the computing time driver side increases, and not the computing time executor side? And this driver computing time is not shown in job processing ui?

If it's the case how can correct it?

Exemple that show the differences between job and batch processing time

1
apache-sparkspark-streaming
Can you see in the tasks view, what task(s) are taking longer? Also, without the code, you will be receiving mostly guesses rather than answers. - maasg
I have edited the post for helping you to suggest an answer. The picture with the batch show some jobs like 44808 and 44797 use cassandra and reduceByKey ( that use checkpointing) - crak

1 Answers

1
votes

I finally found the solution to my problem.

I had this code in the function that add filter and transform to my rdd.

TypeConverter.registerConverter(new SomethingToOptionConverter[EventCC])
TypeConverter.registerConverter(new OptionToSomethingConverter[EventCC])

Because it's call at each batch there is a lot of time the same object inside TypeConverter. And I don't really know how it works Cassandra Spark converter but it's look like to make reflection internaly whit objects. And make slow reflection x time batch make all the processing time of the batch increasing.