spark streaming failed batches

0
votes

I see some failed batches in my spark streaming application because of memory related issues like

Could not compute split, block input-0-1464774108087 not found

, and I was wondering if there is a way to re process those batches on the side without messing with the current running application, just in general , does not have to be the same exact exception.

Thanks in advance Pradeep

1
apache-sparkspark-streaming
Don't you have them already reprocessed during the execution? are you checking the logs for this? - Vale
when I see the spark UI, under streaming, I see some active batches, and when I click on some of them, I see the above exception, that means those batches error-ed out when processing in the streaming application, those batches are the ones I want to deal with , and clean up , and not loose any data because of exceptions.Hope it helps. - Bill
Is your streaming job 'lagging' behind? How is your 'scheduling delay' and 'active tasks'? - maasg
Also, what streaming source are you using? - maasg
scheduling delay is very minimal, there is a problem , and we know why, but how to re process them is the question.The source is apache NIFI - Bill

1 Answers

0
votes

This may happen in cases where your data ingestion rate into spark is higher than memory allocated or can be kept. You can try changing StorageLevel to MEMORY_AND_DISK_SER so that when it is low on memory Spark can spill data to disk. This will prevent your error.

Also, I don't think this error means that any data was lost while processing, but that input block which was added by your block manager just timed out before processing started.

Check similar question on Spark User list.

Edit:

Data is not lost, it was just not present where the task was expecting it to be. As per Spark docs:

You can mark an RDD to be persisted using the persist() or cache() methods on it. The first time it is computed in an action, it will be kept in memory on the nodes. Spark’s cache is fault-tolerant – if any partition of an RDD is lost, it will automatically be recomputed using the transformations that originally created it.