In-order processing in Spark Streaming

Question

Is it possible to enforce in-order processing in Spark Streaming? Our use case is reading events from Kafka, where each topic needs to be processed in order.

From what I can tell it's impossible - each stream in broken into RDDs, and RDDS are processed in parallel, so there is no way to guaranty order.

Answer 1

You could force the RDD to be a single partition, which removes any parallelism.

Answer 2

"Our use case is reading events from Kafka, where each topic needs to be processed in order. "

As per my understanding, each topic forms separata Dstreams. So you should be process each Dstreams one after another.

But most likely you mean you want to process each events your are getting from 1 Kafka topic in order. In that case, you should not depend on ordering of record in a RDD, rather you should tag each record with the timestamp when you first see them (probably way upstream) and use this timestamp to order later on.

You have other choices, which are bad :)

1. As Holden suggests, put everything in one partition
2. Partition with some increasing function based on receiving time, so you fill up partitions one after another. Then you can use zipWithIndex reliably.