How to design API for getting client events from Kafka?

0
votes

I think about architecture pattern, when each client of my service has own kafka consumer.

For example, there is events topic with some replica factor and some number of partitions, which I used for scalability. All events for given client belongs to one partition (I use clientId for partition key).

Each client has own offset. So my API allows to use offset to get client events.

Is it system design fine? Or what is the right API design to get events?

1
javaarchitectureapache-kafkaconsumer
actually better to ask question how to do it ? it's not clear what are you asking – Deadpool
@Deadpool I have service. It has kafka with topic, which contains clients events. I need to develop API for getting these events to clients. – Max
which means you have to consume events from kafka topic is that right? then what is the problem in this? – Deadpool
@Deadpool yea, I have to consume events from kafka topic. The problem is each client has unique offset. So, if I have 100 simultaneous clients, I need 100 kafka consumers, right? What if I will have 1000 online clients? – Max
i did not get this, each client has unique offset. each message in kafka will have offset, what do you mean by client here? According to this (I use clientId for partition key). same client events will got to same partition? what if you have 1000 clients with 100 partitions topic? – Deadpool

1 Answers

0
votes

The question is a little bit vague, so let me state my guesses before going for a proposal.

  1. A single topic where events are published.
  2. These events have to be notified to some clients ( are they mobile apps or something ?)
  3. Current design has one consumer per client, means at least one thread is allocated to for a client.

Concerns with current design

  1. As the number of users increase the number of threads also has to increase, means the approach doesn't scale linearly. Cost increases with number of users.
  2. What if a consumer thread fails? this can cause notification to client to fail.

Proposal

  1. Use Kstreams to consume. Consider kstreams a higher level api for consumption than the consumer api.
  2. Using numthreads property you can configure the number of threads. So a single KStream would act as pool of consumers.
  3. Have a routing logic to find the client and notify.
  4. Trade off : this routing logic adds to the latency.