scaled microservices instances needs to update 1

Question

I have unique problem trying to see what is the best implementation for this.

• I have table which has half million rows. Each row represents business entity I need to fetch information about this entity from internet and update back on the table asynchronously . (this process takes about 2 to 3 minutes) .
• I cannot get all these rows updated efficiently with 1 instance of microservices. so planning to scale this up to multiple instances
• my microservice instances is async daemon fetch business entity 1 at time and process the data & finally update the data back to the table. . Here is where my problem between multiple instances how do I ensure no 2 microservice instance works with same business entity (same row) in the update process? I want to implement an optimal solution microservices probably without having to maintain any state on the application layer.

Answer 1

You have to use an external system (Database/Cache) to save information about each instance.

Example: Shedlock. Creates a table or document in the database where it stores the information about the current locks.

Answer 2

I would suggest you to use a worker queue. Which looks like a perfect fit for your problem. Just load the whole data or id of the data to the queue once. Then let the consumers consume them. You can see an clear explanation here https://www.rabbitmq.com/tutorials/tutorial-two-python.html