Apache camel - EIP design query

Question

I have a EIP design related query.I have a requirement to process csv file by chunks and call a Rest API.After completion of processing of whole file i need to call another Rest API telling processing is complete.I wanted the route to be transacted so i have queue in between in case of end system not available the retry will happen at broker level.

My flow is as below. First flow: csv File->Split by chunk of 100 records->Place message in queue

the second flow(Transacted route): Picks message from queue ->call the rest API

the second flow is transacted.Since iam breaking the flow and it is asynchronous iam not sure how to call to the completion call.I do not have a persistent store to status of each chunk processing.

is there anyway i can achive it using JMS functionality or Camel?

Answer 1

What you can use for your first flow is the Camel Splitter EIP:
http://camel.apache.org/splitter.html

And closely looking at the doc, you will find that there are three exchange properties available for each split exchange:

• CamelSplitIndex: A split counter that increases for each Exchange being split. The counter starts from 0.
• CamelSplitSize: The total number of Exchanges that was splitted. This header is not applied for stream based splitting. From Camel 2.9 onwards this header is also set in stream based splitting, but only on the completed Exchange.
• CamelSplitComplete: Whether or not this Exchange is the last.

As they are exchange properties, you should put them to JMS headers before sending the messages to a queue. But then you should be able to make use of the information at the second flow, so you can know which is the last message.

Keep in mind, though, that it's all asynchronous so the CamelSplitComplete flag doesn't necessarily mean the last message at the second flow. You may create a stateful counter or utilise the Resequencer EIP http://camel.apache.org/resequencer.html to deal with the asynchronicity.