Does producer and consumer need to specify partition

Question

We are planning to have a producer (Java rest api) And consumer (Java client ) running in multiple machines and all the consumers belong to same consumer group.

1) In this case if I do not specify partition during producer publishing messages in which Kafka writes into random/default partition ...

2) While consumer retrieving messages ,if I don't mention partition I'd and just pass the topic name ..and commit the offset to Kafka server..

Is this a good approach if I have one consumer group???

3)do I need to have Kafka rest proxy ,as per my understanding it's not required as I'm using Java client ..please correct me if I'm wrong ...please lemme know what benefit I'll get if I have to use Kafka rest proxy here....

Answer 1

From the producer perspective, if the record doesn't have any key the producer will insert the records in a round-robin way between the partitions in a topic. From the consumer perspective, you don't need to specify the partition if any consumer can handle any record from that topic, if you want actually you don't even have to commit the offset, the client can do it for you with auto.commit.enable=true.

Yes, if you have 1 consumer group you can add multiple consumers and balance the load between the group (this depends on the number of consumers in the group and the number of partitions).

3) You don't need kafka rest proxy when using the java client, besides (If I am not mistaken) the kafka rest proxy has a dependency on the schema registry, so you would have to run both.