I have configured a two node Kafka cluster on AWS, and am currently testing its performance attributes.
I used the kafka-consumer-perf-test.sh to read 50 million messages from a Kafka topic using a single thread.
I observed the following while testing consumer throughput.
Observation 1
Single consumer on a m4.large EC2 instance - Read throughput 40.2MB/sec
Three consumers on three seperate m4.large EC2 instances - Individual read throughput - 40.25MB /sec.
No disk reads or writes were reported on the Kafka broker side ( running on two seperate m4.2xlarge EC2 instances backed by 2 EBS volumes )
I ran the tool again after some time.
Observation 2
Three consumers on three seperate m4.large EC2 instances - Individual read throughput dropped to - 34.25MB /sec.
In this case I observed a considerable rate of disk reads in Kafka broker nodes.
I would appreciate it if you could help me clarify the following.
In observation 1, since I did not observe any disk reads, could all data have been fetched from the Memory ( where it is cached ) ?.
In observation 2, I assume performance dropped due to the disk reads. Although it read from the disk, still enough free memory was available as revealed by nmon reports.
What could have been the reason for consumers to read from disk instead of memory? For how long does data produced by producers stay in the cache?
- I assume maximum 40MB/s read throughput is due to the network bandwidth limitation for m4.large EC2 instance. Is this assumption correct?
Thank you.
