How do I implement Event Sourcing using Kafka?

Question

I would like to implement the event-sourcing pattern using kafka as an event store. I want to keep it as simple as possible.

The idea:

My app contains a list of customers. Customers an be created and deleted. Very simple. When a request to create a customer comes in, I am creating the event CUSTOMER_CREATED including the customer data and storing this in a kafka topic using a KafkaProducer. The same when a customer is deleted with the event CUSTOMER_DELETED.

Now when i want to list all customers, i have to replay all events that happened so far and then get the current state meaning a list of all customers.

I would create a temporary customer list, and then processing all the events one by one (create customer, create customer, delete customer, create customer etc). (Consuming these events with a KafkaConsumer). In the end I return the temporary list.

I want to keep it as simple as possible and it's just about giving me an understanding on how event-sourcing works in practice. Is this event-sourcing? And also: how do I create snapshots when implementing it this way?

One big code smell I see here is that you want to persist data in a queue manager. Queues are not meant to persist data. — Itay Moav -Malimovka
yes they are not but its possible and I would like to try :) jsut not sure if I am on the right track. — M4V3N
@ItayMoav-Malimovka But Kafka is meant to persist data for a configurable amount of time (including forever), so...? — OneCricketeer

OneCricketeer OneCricketeer · Accepted Answer · 2019-01-20T00:15:03

when i want to list all customers, i have to replay all events that happened so far

You actually don't, or at least not after your app starts fresh and is actively collecting / tombstoning the data. I encourage you to lookup the "Stream Table Duality", which basically states that your table is the current state of the world in your system, and a snapshot in time of all the streamed events thus far, which would be ((customers added + customers modified) - customers deleted).

The way you implement this in Kafka would be to use a compacted Kafka topic for your customers, which can be read into a Kafka Streams KTable, and persisted in memory or spill to disk (backed by RocksDB). The message key would be some UUID for the customer, or some other identifiable record that cannot change (e.g. not name, email, phone, etc. as all this can change)

With that, you can implement Interactive Queries on it to scan or lookup a certain customer's details.

How do I implement Event Sourcing using Kafka?

2 Answers