Storm for batch like processing

Question

I am new to Storm and have few basic questions. My use case for storm is both stream and batch processing.

Use Case #1: Storm topology takes in the tuples as stream and processes it.

Use Case #2: Storm topology should take in the tuples as a batch of tuples and process it.

I'm using Kafka as the queuing mechanism to feed Storm topology.

Question: Is there a way, where I can tell that the a particular tuple is the end of the stream and storm should tell me when the processing of all the tuples is finished?

Is Storm not the correct framework to do this, as it is meant for stream processing (Use case #1). Would Storm Trident help for use case #2?

Answer 1

You cannot tell Storm, that a tuple is the last of a stream. However, if you know that you just emitted the last tuple from your Spout, you can set an internal flag for yourself and furthermore wait until you received all acks in the Spout. When all acks are received, you know that all tuples got processed completely by Storm.

For question 2, it is not clear to me, what you mean by "do the same processing"? It seems, you want to process the same data twice in different modes (or did I understand something wrong)? Why do you distinguish between "stream" and "batch" case? What is the different semantics you want to get? And what do you mean by "take in the tuples as a batch of tuples". Do you know that you have an finite data stream? Do you want to put all tuples into a single batch? Or do you want to do some micro-batching?

For micro-batching, Trident would be useful. If you have a real batch job, Storm is no good fit. For this, you might want to check out Apache Flink (disclaimer, I am a committer to Flink) or Apache Spark. Both are hybrid systems supporting batch and streaming. Depending on your needs on streaming semantics the one or other might be the better fit. Spark provides micro-batching to emulate streaming while Flink does real streaming.