How can i design a bridge from a legacy CRUD oriented app to a CQRS and Event sourcing system?

Question

I was asked to implement CQRS/Event sourcing patterns into a legacy web application, in order to prepare to migrate it from a monolithic/state oriented model to a distributed, service oriented app.

I have some questions on how i can design a Domain oriented code bundle that would connect the legacy entities strongly coupled to database, with a new Event sourced model.

The first things i did was:

1. writing a small "framework" for CQRS/ES, with classes like AggregateRoot, DomainEvent, Command, Handlers, Messaging, Eventstore, AggregateIds, etc.
2. trying to group and "migrate" the legacy Entities into some Aggregates to reconstructe all the history and states of the app into EventSoourced Aggregates
3. plug some Commands dispatching in the old controllers in order to let the app work as is, but also to feed the new CQRS/ES system on the side.

The context:

The legacy app contains several entities, mapped to database, that hold the model layer. (Our domain is Human resources (manpower). Let's say we have those existing entities:

1. Worker, with various fields and related entities (OneToOne, OneToMany), like

   • name
   • address 1-1
   • competences 1-N

2. Society, in which worker works, with various fields and related entities (OneToOne, OneToMany), like

   • name
   • address 1-1
   • hours

3. Contract, with various fields and related entities (OneToOne, OneToMany), like

   • address 1-1
   • Worker 1-1
   • Society 1-1
   • documents 1-N
   • days 1-N
   • hours
   • etc.

From this legacy model, i designed a MissionAggregate that holds:

• A db independant ID, like UUID
• some Value objects: address, days (they were an entity in the legacy model, they became VOs here)

I also designed a WorkerAggregate and a SocietyAggregate, with fields and UUIDS, and in the MissionAggregate i added:

• a reference to WorkerAggregate's UUID
• a reference to SocietyAggregate's UUID

As i said earlier, my aim is to leave the legacy app as is, but just introduce in the CRUD controller's methods some calls to dispatch Commands to the new CQRS system.

For example:

After flushing newly created Contrat in bdd, i want to dispatch a "CreateMissionCommand" to the new command bus.

It targets the appropriate Command Handler, that handles all the command's datas, passes it to a newly created Aggregate with a new UUID and stores "MissionCreatedDomainEvent" in the EventStore.

The DomainEvent is indexed with an AggregateId, a playhead, and has a payload wich contains the fields necessary to be applied to and build the MissionAggregate.

The newly Contract created in the app has now its former lifecycle, as usual, with all the updates that the legacy app does on it. But i also need to reflects all those changes to the corresponding EventSourcedAggregate, so everytime there is a flush in database in the app, i dispatch a Command that translates the "crud like operations" of the legacy app into a Domain oriented /Command oriented pattern.

To sum up the workflow is:

1. A Crud legacy operation occurs and flushes some changes on the Contract Entity
2. In just a row of code in the controller, i dispatch a command builded with necessary fields (AggregateId of the MissionAggregate...that i need to have stored somewhere..see next problems) to the Domain command bus , so that the impact on the existing code base is very low.
3. The bus passes the command to the corresponding command handler
4. The handler loads the aggregate and applies the changes it by calling the appropriate Aggregate method
5. then after some validation, the aggregate raises and stores the appropriate event

My problems and questions (some of them at least ;)) are:

1. I feel like i am rewriting all big portions of the legacy app, with the same kind of relations between the Aggregates that i have between the Entities, and with the same type of validations, checks etc.

2. Having references, to both WorkerAggregate and SocietyAggregate UUID in MissionAggregate implies that i have to build those aggregate also (hence to dispatch commands from legacy app when the Worker and Society entities are flushed). Can't i have only references to Worker's entity id and Society's entity id ?

3. How can i avoid having a eternally growing MissionAggregate ? The Contract Entity is quite huge, it has a looot of fields that are constantly updated (hours, days, documents, etc.) If i want to store all those events, i need to have a large MissionAggregate to reflect all those changes; and so i need to have a tons of CommandHandlers that react to all the Commands of add, update, etc that i am going to dispatch from the legacy app.

4. How "free" is an Aggregate from the Root entity it is supposed to refer to ? For example, a Contract Entity needs to relate somewhere to it's related Mission Aggregate, like for example when i want to dispatch a Command from the app, just after the legacy code having flushed something on the Entity. Where to store this relation ? In the Entity itself, in a AggregateId field ? in the Aggregate, should i have a ContratId field ? Or should i have some kind of Mapping Table somewhere that holds the relationship between Contract ID and MissionAggregate ID ?

5. What to do with the past ?should i migrate all the existing datas through a script that generates Aggregates and events on all the historical data ?

Thanks in advance for your time.

Answer 1

You have a huge task ahead of you, let's try to break it down.

It's best to build this new part of the system in isolation from the legacy codebase, otherwise you're going to have your hands tied in every turn of the way.

Create a separate layer in your project for these new requirements. We're going to call it "bubble" from now on. This bubble will be like a greenfield project, with its own structure, dependencies, etc. There will be no direct communication between the bubble and the legacy; communication will happen through another dedicated translation layer, which we'll call "Anti-Corruption Layer" (ACL).

ACL

It is like an API between two systems.

It translates calls from the bubble to the legacy and vice-versa. Its purpose is to prevent one system from corrupting or influencing the other. This way you can keep building/maintaining each system independently from each other.

At the same time, the ACL allows one system to consume the other, and reuse logic, validations, rules, etc.

---

To answer your questions directly:

1. I feel like i am rewriting all big portions of the legacy app, with the same kind of relations between the Aggregates that i have between the Entities, and with the same type of validations, checks etc.

With the ACL, you can resort to calling validations and reuse implementations from the legacy code. This will allow you time to rewrite things as needed or as possible.

You may not need to rewrite the entire system, though. If your goal is to implement CQRS and Event Sourcing and you can achieve this goal by keeping most or part of the legacy system, I would say you do it. Unless, of course, one of the goals is to completely replace the old system. Otherwise, keep it; write as less code as possible.

Suggested workflow:

• Keep the CQRS and Event Sourcing system in the bubble
• Do not bring these new frameworks into legacy
• Make the lagacy Controller issue method calls to the ACL
• The ACL will convert these calls into Commands and dispatch them
• Any events will be caught by your Event Sourcing framework
• Results will be persisted to the bubble's database

The bubble's database can be a different schema in the same database or can be a different database altogether. But you'll have to think about synchronization, and that's a topic of its own. To reduce complexity, I recommend a different schema in the same database.

2. Having references, to both WorkerAggregate and SocietyAggregate UUID in MissionAggregate implies that i have to build those aggregate also (hence to dispatch commands from legacy app when the Worker and Society entities are flushed). Can't i have only references to Worker's entity id and Society's entity id?

3. How can i avoid having a eternally growing MissionAggregate ? The Contract Entity is quite huge, it has a looot of fields that are constantly updated (hours, days, documents, etc.) If i want to store all those events, i need to have a large MissionAggregate to reflect all those changes; and so i need to have a tons of CommandHandlers that react to all the Commands of add, update, etc that i am going to dispatch from the legacy app.

You should aim for small aggregates. Huge aggregates are likely to degrade performance and cause concurrency problems.

If you anticipate having a huge aggregate, it is best to rethink it and try to break it down. Ask what fields/properties change together - these are possibly a different aggregate.

Also, when you speak about CQRS, you generally lean towards a task-based way of doing things in your system.

Think of a traditional web application, where you have a huge page with lots of fields that are all sent to the server in one batch when the user saves.

Now, contrast it with a modern web app where the user changes small portions of data at each step. If you think about your system this way you'll find those smaller aggregates.

PS. you don't need to rebuild your interfaces for this. If your legacy system has those huge pages, you could have logic in the controllers to detect which fields were changed and issue the appropriate commands.

4. How "free" is an Aggregate from the Root entity it is supposed to refer to ? For example, a Contract Entity needs to relate somewhere to it's related Mission Aggregate, like for example when i want to dispatch a Command from the app, just after the legacy code having flushed something on the Entity. Where to store this relation ? In the Entity itself, in a AggregateId field ? in the Aggregate, should i have a ContratId field ? Or should i have some kind of Mapping Table somewhere that holds the relationship between Contract ID and MissionAggregate ID?

Aggregates represent a conceptual whole. They are like atoms, indivisible things. You should always refer to an aggregate by its Root Entity Id, and never to a Child Entity Id: looking from the outside, there are no children.

An aggregate should be loaded as a whole and persisted as a whole. One more reason to have small aggregates.

An aggregate can be comprised of a single entity. Or it can have more entities and value objects, forming a graph, but one entity will be elected as the Root and will hold references to its children. Child entities and value objects should not hold references to their parents. The dependency is not bi-directional.

If Contract is an entity inside the Mission aggregate, the Contract should not have a reference to its parent.

But, if your Contract and Mission are different aggregates, then they can reference each other by their Ids.

5. What to do with the past? Should i migrate all the existing datas through a script that generates Aggregates and events on all the historical data?

That's a question for the business experts. Do they need it? If they don't, then don't implement it just for the sake of doing so. Every decision you make should be geared towards satisfying a business need and generating real value for it, considering the costs and tradeoffs.

Some people say that code is a liability, not an asset, and I aggre to some extent: every line of code you write needs to be tested and supported. Don't write any code that is not really necessary.

---

Also, have a look at this article about the Strangler Pattern, which shows how to migrate a legacy system by gradually replacing specific pieces of functionality with new applications and services.

If you have a chance, watch this course at Pluralsight (paid): Domain-Driven Design: Working with Legacy Projects. The author presents practical approaches for dealing with this kind of task.

I hope this has given you some insight.

Answer 2

I don't want to spoil your game. Everybody knows how cool it is to rewrite something from scratch. It's a challenge, it's fun, it's exciting. However...

migrate it from a monolithic/state oriented model to a distributed, service oriented app

CQRS/Event Sourcing won't solve any of your problems and it won't help you distribute the app in any reasonable way. If you just generate events on the CRUD operations you'll have a large tangled mess of dependencies between each part. Every part that needs data will have to call a couple of "services" (i.e. tables) to get it, than push data elsewhere, generate events1 that some other parts will react to. It will be a mess. Usually this is called a distributed monolith.

This is also the reason you already see problems with it. These problems won't go away, because you are essentially building the same system in the same way, but this time it'll be more complex.

Where to go from here

The very first thing is always: have a clear goal. You want a service oriented architecture you said. Why? Are there parts that need different scaling, different resources? Are they managed by different teams with different life-cycles? Etc.? Maybe you already have all this, I don't know, but if not, that's your first task.

Then. The parts you do want to pull out can't be just CRUD things. Those will not be independent, so whether your goal (see point above!) is scaling or different team, you won't reach your goal! To be independent you'll have to pull out the behavior with the data, and in a way that the service can operate on its own.

You can't just throw buzzwords at it and hope for the best. I'd suggest to just ignore all the hype and buzzwords and think about the goal you want to reach.

For example: I need a million workers to log their time in under 10 minutes total. So that means I need a "service" to enable worker to log their time with a web interface. So let's create that as a complete independent piece with its own database so it can be scaled to a 100 nodes when it needs to be. Export data to billing automatically every hour or so.