We are using Apache Kafka 2.2 version with 5 brokers. We are getting 50 millions of events per a day but we are hitting high kafka cpu usage. We are using default producer/consumer/broker settings.
I have some questions about the performance;
We have different kafka-streams applications which do aggregation, or join operations to carry enriched messages. All of our kafka-streams applications consist of these settings:
- exactly-once:true
- min-in-sync replicas:3
- replication factor: 3
- topic partition: ~50-~100 (Differs on topology)
and of course there may be internal topics for topologies. We are scaling up our worker machines into at least 5 applications. So each instance looks for one thread for one partition strategy most of the time.
However, beside the topology optimization, is there something we can do on the default settings?
Beside the kafka-streams application we are using spring-kafka producer/consumer. But like I said we are using default settings. For example we are producing events one by one on the producer side.
Our throughput is not fast enough and we have high cpu usage. If we close the some of our kafka-streams application, brokers load are decreasing. So my question is;
Does the in-sync-replicas with exatly-once true and replication factor of 3 applies too much load on the brokers ? I do not want to lose or deduplicate my messages on prod environment so my streams applications must consist of exaclty-once true, but the spring applications do well without kafka-streams.
I want to decrease load the overall cpu usage of the broker in our system.
If I use the batch producer on the producer side, decrease min-in-sync replicas for my fault tolerable workers, does my cpu usage decrease ?
I can't think the optional way. Any idea would help me why my brokers CPU usage is very very high like around %80-90 day time.
What can cause the high cpu usage on the brokers ?
