CQRS architecture optimizations and variations

Question

A Command Query Responsibility Separation / Event Sourcing architecture is a clear fit for a project I'm starting, which will see around a billion financial transactions a year related to people's health insurance cover. The key benefits are audit history, scalability, enforcing an asynchronous-compatible UI across a number of teams, splitting these transactions from the read database, easing the transmission of state to intermittently connected field offices via event queues, and coping with significant business logic change throughout the system lifetime.

However there are areas where CQRS/ES is going to be problematic, for example assigning a numeric ID to 100mn people, user security where eventual consistency is not acceptable. There are also areas of the system that are CRUD in nature and do not benefit from CQRS/ES. Finally we will have a large number of developers in different teams and companies and it would be good to have areas that do not require CQRS/ES competence. Is it possible to employ a hybrid approach, where some areas are not event sourced? Could we just synchronize the relevant tables on both read and write sides?

Do CQRS-architected aggregate entities simplify snapshot cache invalidation? Any event that updates an aggregate entity that might be cached could be listened for by an invalidator, and given aggregate entities are coarser-grained than relational entities and we can distinguish write events does this problem become solvable?

I'm expecting around a billion events a year and a need to track around 4 years of history. Can we snapshot and archive older events?

Are there degrees of event sourcing? For example one online store system AddLineItem event might include line item price per unit, but rely on the read side to pull and render the product name on the invoice. Another online store might include the name in the event data. How do you choose what to include in the event? In health insurance it might limit what 'what if' analyses can be run - if we haven't included the insured person's age we can't feasibly simulate policies that require it?

Is there an interesting way to model events about events? For example an administrator enters into the system that a product's price will change at some date in the future. I suppose the snapshot would be a price timeline. Could we instead add a post-dated ProductPriceChanged event? Could we fake such events when running 'what if' scenarios? (Such aggregates would have to be rarely changed, to avoid version number and concurrency detection issues.)

CQRS/ES is often claimed to make the system easier to adapt to future business process change. I understand the argument that commands listing events in ubiquitous language makes discussing and reconfiguring them easier, and event sourcing removes some of the rigidities of the RDBMS model. But won't any changes within an event will break the replay of events? With systems subject to change won't you end up with many versioned events? Eg, in an online store assessing whether a client is a Gold Card Holder by changing criteria? Can you snapshot everything? How would you post-date these changes? Similarly do you have to be careful with dependency injection that none of the dependencies injected can affect business logic, as otherwise you break replay?

Any idea why it is associated with the .NET world, with less popularity in other areas of the industry?

Huge thanks even for just reading.

Answer 1

Is it possible to employ a hybrid approach, where some areas are not event sourced?

Of course.

Could we just synchronize the relevant tables on both read and write sides?

That sounds like a bad idea in many cases.

Do CQRS-architected aggregate entities simplify snapshot cache invalidation?

Not much? The notion that there is an event that signals a write has occurred to the book of record, which can be used to invalidate caches, isn't a particularly CQRS idea. It's not any simpler -- it's just that the extra work was in scope anyway.

Can we snapshot and archive older events?

Yes but... it's usually easier to think in terms of the history of a long lived entity being broken into shorter episodes, and rolling over state from one episode to the next -- for instance, consider rolling over the ledgers at the end of a financial period. Then you archive the histories of any aggregate that is end of life.

How do you choose what to include in the event?

Review the state needed by this aggregate to establish/maintain/restore the business invariant. Everything else can go in the wash. That often means that reports (read models) are assembled from multiple aggregates, and possibly documents as well.

Is there an interesting way to model events about events?

Events about events is messy. Events about processes are awesome.

Could we instead add a post-dated ProductPriceChanged event?

Wrong spelling -- try PriceChangeScheduled. Note: modeling time is important; the domain model shouldn't notice the passage of time unless the outside world mentions it.

But won't any changes within an event will break the replay of events?

No, but there are disciplines you need to maintain about the event representations to ensure that this is so. Greg Young is writing Versioning in an Event Sourced System as an e-book.

The quick and dirty -- fields in the schema are optional; you can add them, or remove them, but you never change their meaning. Consumers provide a default value for anything they want to read, and "must ignore" entries they don't understand.

With systems subject to change won't you end up with many versioned events? Eg, in an online store assessing whether a client is a Gold Card Holder by changing criteria?

Not clear to me which question this is. There may be several representations of an event (depending on what schema is expected and which default values are considered), but it's still just one event. Decisions made by the system are documented by events, so versioning over time doesn't really enter into it.

Similarly do you have to be careful with dependency injection that none of the dependencies injected can affect business logic, as otherwise you break replay?

The business logic all lives in the domain model, and the domain model lives in the center of the onion; detached from the real world. So you shouldn't be injecting any dependencies that introduce side effects in the real world. Those are typically handled asynchronously (we successfully saved these events, and therefore side effects can be scheduled).

Any idea why it is associated with the .NET world, with less popularity in other areas of the industry?

People. Udi Dahan and Greg Young both have a .NET background. There's also popularity in PHP, because Mathias Verraes has that background.

how would you suggest I persist non-ES aggregate entities?

Document store? RDBMS? Flat file? Polyglot persistence is fine.

I might miss how this answers this question - if the business invariant is an order line item, does this include product name or just price?

Product Id and quantity are probably enough. Maybe a price quote, if you are in a business where the quote might be different from what is listed in the catalog.

I mean if an aggregate's business logic changes but past events still need processing the old way.

One key idea is that the meaning of events should not change; they describe changes in state. So if you find that the "new version of an event" means something different, then you really have a new event. See Young's book.

An aggregates business logic -- deciding how to evolve from one state to the next; that does change. But it doesn't change what state the given aggregate is actually in.

For instance, you might discover that some state shouldn't be reachable. That's business logic -- the aggregate shouldn't write new events to end up in that state. That doesn't affect in any way aggregates that are currently in the unreachable state; they are still there, because that's where the history put them. You move them out of that state by giving them more events.