How do you get data from a reactive microservices system?

0
votes

I'm designing a reactive microservices system, where communication between services – request processing pipeline – is async (using kafka). The system will manage cluster of VMs. Panning to use Spring WebFlux as a framework.

For example, request to create new VM might look like this:

  1. API Gateway
  2. Security Validation
  3. Create VM in the userspace
  4. Schedule resources for VM
  5. Send notification to the user once succeeded/failed

All the steps are sequential, yet async, thanks to kafka. Steps are processed by separate microservices.

Besides that, client will (might) want to issue GET requests, like get list of VMs, or info about cluster state. And I just don't know how. I can think of the following solutions:

  1. Make (synchronous) GET requests from the API gateway to other services, aggregate data and return.
  2. Notifications to the user already have all the required data, so I can aggregate all user-queryable data in dedicated service (save to the database) and forward GET requests from API Gateway there.
1
asynchronousapache-kafkaarchitecturemicroservicesreactive-programming

1 Answers

1
votes

I would actually suggest not using a pipeline across services like this and have a service that manages this saga and tracks the status of the saga.

  1. API gateway forwards request to saga management service
  2. Saga management service notifies security validation service
  3. Security validation service publishes event
  4. Based on event from 3, saga management may notify VM creation service
  5. VM creation service creates VM in userspace, publishes event
  6. Based on event from 5, saga management may notify resource scheduler service
  7. Resource scheduler service schedules resources, publishes event
  8. Based on event from 7, saga management sends notification

The saga manager can respond to failures (e.g. by rolling back VM creation if resources couldn't be scheduled or poking services to determine if it makes sense to consider them down) and effectively interpret things to publish extra-rich events for other services (e.g. a service that can list VMs is probably interested in events involving VM creation and resource scheduling, not just the notifications) and respond to interactive requests.