Lets say you have a service A that is part of large microservice architecture, where those services communicate between each other either via REST APIs or via messaging where some broker is involved (RabbitMQ). Service A expose REST endpoint that needs to communicate with some 3rd party service (so with service that is not in our architecture) and to create some stuff there, when service A receives a response from 3rd party that everything went good there, it should persist some data from that response in its own database.
What would be the best way of covering following issues, having in mind that 3rd party does not provide any idempotence mechanism.
Creation on 3rd party side went good, but DB write failed in service
A. This would lead to inconsistent state, you created something on 3rd party side, but you don't have needed data about it in your own database.You received timeout from 3rd party, so you can't just repeat the call, cause they are not providing any idempotence mechanism. If you repeat the call, potentially you can end with two (or more) created resources instead of one.
Problem number 1. could be solved having whatever retry mechanism that could retry DB call for whatever number of times. Problem with that approach is that, if service A instance that is repeating the DB call goes down suddenly.
Presumably, better approach would be that service after successful creation on 3rd party side, publish a RabbitMQ message about successful creation. This service would listen to that message and it could do DB call when it receive the message. Having nice retry mechanism and taking benefit of ACKing messages, one could solve the issue about, What if service instance goes down suddenly. So in this solution service is publisher and consumer of its own message(s). Any better idea? This solution will also introduce eventual-consistency, because API caller (guy who is calling service A endpoint) will receive response right after successful creation on 3rd party side, but before anything is persisted in service database (what API client need actually)
What about timeout problem? How one would handle timeouts from 3rd party in this case?. I don't see anything better than issuing GET calls to check do they created something. Still again that GET call can fail, but it can be repeated until it succeed. Here also marginal use case is what if service goes down in time of issuing GET calls.
