Benchmarking Kafka - mediocre performance

Question

I'm benchmarking Kafka 0.8.1.1 by streaming 1k size messages on EC2 servers.

I installed zookeeper on two m3.xlarge servers and have the following configuration:

dataDir=/var/zookeeper/                                                                    
clientPort=2181
initLimit=5
syncLimit=2
server.server1=zoo1:2888:3888                                
server.server2=zoo2:2888:3888

Second I installed Single Kafka Server on i2.2xlarge machine with 32Gb RAM and additional 6 SSD Drives where each disk partitioned as /mnt/a , mnt/b, etc..... On server I have one broker, single topic on port 9092 and 8 partitions with replication factor 1:

broker.id=1
port=9092
num.network.threads=4
num.io.threads=8
socket.send.buffer.bytes=1048576
socket.receive.buffer.bytes=1048576
socket.request.max.bytes=104857600
log.dirs=/mnt/a/dfs-data/kafka-logs,/mnt/b/dfs-data/kafka-logs,/mnt/c/dfs-data/kafka-logs,/mnt/d/dfs-data/kafka-logs,/mnt/e/dfs-data/kafka-logs,/mnt/f/dfs-data/kafka-logs
num.partitions=8
log.retention.hours=168
log.segment.bytes=536870912
log.cleanup.interval.mins=1
zookeeper.connect=172.31.26.252:2181,172.31.26.253:2181
zookeeper.connection.timeout.ms=1000000
kafka.metrics.polling.interval.secs=5
kafka.metrics.reporters=kafka.metrics.KafkaCSVMetricsReporter
kafka.csv.metrics.dir=/tmp/kafka_metrics
kafka.csv.metrics.reporter.enabled=false
replica.lag.max.messages=10000000

All my tests are done from another instance and latency between instances is less than 1 ms. I wrote producer/consumer java client using one thread producer and 8 threads consumer, when partition key is a random number from 0 till 7. I serialized each message using Json by providing custom encoder.

My consumer producer properties are the following:

metadata.broker.list = 172.31.47.136:9092
topic = mytopic
group.id = mytestgroup
zookeeper.connect = 172.31.26.252:2181,172.31.26.253:2181
serializer.class = com.vanilla.kafka.JsonEncoder
key.serializer.class = kafka.serializer.StringEncoder
producer.type=async
queue.enqueue.timeout.ms = -1
batch.num.messages=200
compression.codec=0
zookeeper.session.timeout.ms=400
zookeeper.sync.time.ms=200
auto.commit.interval.ms=1000
number.messages = 100000

Now when I'm sending 100k messages, I'm getting 10k messages per second capacity and about 1 ms latency.

that means that I have 10 Megabyte per second which equals to 80Mb/s, this is not bad, but I would expect better performance from those instances located in the same zone.

Am I missing something in configuration?

To take the producer side performance out of the equation (which historically has been quite bad), try running rdkafka_performance -P .. from librdkafka and likewise with the consumer do the same with rdkafka_performance -C ... A rule of thumb on the consumer performance is that it should be around the speed of the disk, or network, whichever is slower, as long as you are not using compression. — Edenhill

Peter Lawrey Peter Lawrey · Accepted Answer · 2014-12-12T17:09:48

I suggest you break down the problem. How fast is it without JSon encoding. How fast is one node, without replication vs with replication. Build a picture of how fast each component should be.
I also suggest you test bare metal machines to see how they compare as they can be significantly faster (unless CPU bound in which case they can be much the same)

According to this benchmark you should be able to get 50 MB/s from one node http://kafka.apache.org/07/performance.html

I would expect you should be able to get close to saturating your 1 Gb links (I assume thats what you have)

Disclaimer: I work on Chronicle Queue which is quite a bit faster, http://java.dzone.com/articles/kafra-benchmark-chronicle

Benchmarking Kafka - mediocre performance

2 Answers