Storm topology transaction and Database transaction

4
votes

I am using storm, A topology has a bolt "A" which insert record in database and emit to another bolt "B". If a tuple bolt "B" execute failed, How to rollback the tuple effected on database when bolt "A" executed? Anyone can help me? THX

1
javareal-timeapache-storm
You need to use Trident which is an abstraction on top of Storm to guarantee once-only processing. - Jim Garrison
Will that guarantee that the tuple is only inserted in the database once? That is, can you be sure that each bolt only executes each tuple once? Or does "exactly once" just mean that once a tuple has been acked by all bolts, it will never be replayed? - torbonde
@JimGarrison yep, if B failed the tuple will be retried from spout and Trident guarantees the tuple can't insert record again in A? but before retrying done, database may have dirty data? - user1221244
If you need to ensure database consistency (i.e. a transaction) across more than one bolt I think Storm may not be adequate. I'm not a Storm expert but everything I've read indicates bolts that update the database must be transactionally self-contained. - Jim Garrison
@JimGarrison You can get around those sorts of things by building your own transaction layer on top of the DB. I've done this lots of times to ensure database consistency across multiple bolts in a topology. - G Gordon Worley III

1 Answers

1
votes

To understand how storm deals with transaction take a look at their Transactional topology documentation . Its nicely written and should give you a good concept on the same.

Now instead of managing the transaction using normal storm you can go for trident which a top level abstraction build on top of storm and allows a better approach towards transactional functionalities. The documents are nice and describe the approach in details.

Also look here to know how storm guarantees messages to be processed fully.