Sudden Increase in Scheduling delay for spark streaming Job Without change in other parameters

5
votes

I have a spark streaming job running in production with 1 sec batches. I use CDH 5.5 Spark 1.5. We use Kafka Create Directstream. We have enabled Back Pressure. We dont want to sue dynamic allocation So the job executed with fix number of executor.

From the below image i can see that these is sudden increase in the scheduling delay from 13.50 But during the same time i dont see any dealy in the processing time.

  1. What would be possible reasons for increase in the scheduling time whne processing times are same.
  2. Does other job loads in the cluster effect the current streaming job. In my understanding it should not be the case because the executors for streaming are pre-allocated and are already running

Any thoughts?

enter image description here

1
apache-sparkspark-streaming
Were you able to resolve this? I'm observing a very similar issue in my Spark streaming app that reads its inputs from Kafka (using the DirectKafkaStream approach) - jithinpt
We didnt had specific solution for this. Playing around with Batch Duration and Executor Memory and Vcores solved our issue. - kalyan chakravarthy

1 Answers

0
votes

This is indeed a strange issue at first, but lets get to this point Does other job loads in the cluster effect the current streaming job. The answer is that the cpu share will be affected if another process is starting to run on the same cluster and could lead to contention at which point you would see waiting. Are you running spark in a container by any chance? Its also difficult to fully comprehend your problem as I'm not aware of how you're setting up your cluster.