3
votes

I'm planning a model for microservices using event sourcing. To achieve a high scalability and high throughput handling capacity, I'll use Kafka as a message broker for the microservices.

At this point, I have questions about the implementation of the model to be able to have the benefits of Kafka topic and partition. There are some requirements that my model needs to fit:

  1. Microservices must ingest data from the message broker (POST/PATCH/PUT/DELETE)
  2. Data consistency is mandatory, if entity A needs previous existence of entity B, then must exists only records of entity A that points to valid records of entity B
  3. The microservices can't couple their domains (refering to DDD)
  4. The system must handle high thrgoughput of read and writes operations

Well, with this requirements in mind, I've endend up with a model that:

  1. Use an Elasticsearch database that have the current valid state
  2. All write requests are received by a "Microservice Gateway" that:
    1. Validate the request and the new state that is requested
    2. Produce a write operation in the message broker
    3. Receives state change events from the message broker
    4. Responds to requester with the new state changed
  3. All write operations are consumed by a "Microservice Processor" that:
    1. Handles all the business logic and data denormalization
    2. Updates the state in the Elasticsearch database
    3. Produce a state change event in the message broker
  4. All read requests are handled by the "Microservice Gateway" that:
    1. Searchs at the Elasticsearch database for the requested resource's records
    2. Responds to the requester with the results

My questions:

  1. This model does some coupling of the domains? The Gateway is validating subresources ID's to ensure data consistency in one Elasticsearch database (the case of A pointing to B).
  2. I don't know if this model fits reports requests, there are some complex reports that process a lot of data with input parameters from the user and from his point of view, the operation must be "synchronous" (request/response REST)
  3. Is the validation of requests/new state a part of the business logic (related to DDD)? If so, my model is incorret to separate them into two microservices?

EDIT

My new proposal for my client:

  1. Instead of having a gateway acting as a part of the microservice, let gateway only for: routing (microservice registry), balancing and auth stuffs (calling a dedicated microservice for authentication/authorization)
  2. The microservices hold their own data/database, consistency is ensured by event sourcing architeture
  3. Reports: if it's about one domain, can be a resource at the microservice that holds the data, if more than one domain is required, another microservice will provide the report
1
I added a few comments in my answer.Oswin Noetzelmann

1 Answers

2
votes

This model does some coupling of the domains? The Gateway is validating subresources ID's to ensure data consistency in one Elasticsearch database (the case of A pointing to B).

You answered your own question with the following sentence:

All read requests are handled by the "Microservice Gateway"

If that is true your system does not have a domain separation at all. In fact you have a god-Gateway that needs to know about all domains.

A gateway should be constrained to very dedicated purposes that the domain specific services cannot fulfill (e.g. security, routing/proxy, load balancing, consolidation, resilience, etc.).

I don't know if this model fits reports requests, there are some complex reports that process a lot of data with input parameters from the user and from his point of view, the operation must be "synchronous" (request/response REST)

I would see reports as a separate service that ideally uses the existing services to get whatever information is required. Depending on your domain and requirements you may want to have some dedicated non-public endpoints at your services for this or you may want to access the database directly (e.g. with specialized queries on graph databases).

Is the validation of requests/new state a part of the business logic (related to DDD)? If so, my model is incorret to separate them into two microservices?

It depends on the type of validation. For validating correctness from a domain point of view you need the corresponding service. If you want to validate other things other parts of your system could be responsible (logging, authorization, thwarting malicious attempts, ...)

*For your edit:

  1. Instead of having a gateway acting as a part of the microservice, let gateway only for: routing (microservice discovery), balancing and auth stuffs (calling a dedicated microservice for authentication/authorization)

Few things about this:

  1. Typically you don't do the actual service discovery on the API gateway. Instead it should be a function of the cluster orchestration (see how kubernetes does it for you). The gateway would just use the service registry to proxy correctly (for example use the right DNS entries).
  2. The microservices architectural style was invented to allow very large organizations to scale fast growing server applications. It does have drawbacks, especially when setting up for smaller projects that need to have results quickly. Most people from those larger organizations would tell you to start with a monolithic initially and separate out when teams grow too big (by drawing component boundaries you can prepare for that separation in initial development). Even with a "monolithic" service you can still have a modern build/test/deployment infrastructure with CICD and the service can still have horizontal scalability. Since I don't know your domain / project size I just wanted to give that as a general thought.