Spark Stream Dstream RDDs order

0
votes

The documentation says:

Any operation applied on a DStream translates to operations on the underlying RDDs. For example, in the earlier example of converting a stream of lines to words, the flatMap operation is applied on each RDD in the lines DStream to generate the RDDs of the words DStream. This is shown in the following figure.

Spark Doc

My questions is Does the RDDs in the Dstream get processed in order. For instance Let say you put the Dstream into one Partition: 

orders.map(o => (o.buy, 1L)).repartition(1)

In other work, you do the work in parallel and then repartition everything in one partition, so as to process it sequencially and maintain the order.

Given that, the RDDS come by interval, i wonder if an RDD that arrive later at t2 can be proceesed before an RDD that arrived earlier at t0.

I just want my Dstream to maintain the global Order.

1
spark-streaming

1 Answers

1
votes

In spark, operations such as map, filter, flatMap, and coalesce ensure order.

But repartition does not guarantee order.

It works the same in streaming.

Use coalesce if you want to reduce the number of partitions to 1.