I'm working on PoC for small process engine based on Camel. Requirements are to have ability to execute set of consequence steps and each of them could potentially take hours to execute. Asynchronous communication style is obvious choice in this case but I'm struggle with getting "process" part right.
When sending message to external system I need to wait for completion. As long as it might take lot of time I'm thinking about stopping processing of concrete step after I sent message and then starting new "job" upon getting completion message back. So literally processing of each step will be handled as a Camel route starting at same JMS queue and then content-based router will decide which concrete logic should be executed based on headers of message or its content.
The problem, however, is to how to avoid potential of messages loss. For example, at concrete step I'm sending message and stop processing. The external system for some reason did not process message thus my system does not receive any notification. This means that process is stuck unless some other component will generate message to wake it up.
Also as long as system could be shutdown at any point of time I have to build in logic to continue processing messages after restart (which implies some kind of message persistence, redelivery and transaction management strategy).
All this issues add up thus I'd like to ask experience Camel champions to provide suggestions on how to design such kind of logic using Camel. I know that dedicated BPM product or ESB might handle this problem much more easier but I don't want to bloat the solution.
Any advices are welcomed, especially in terms of Camel capabilities that could help in simplification of solution.