Java Producer not recognizing property "Partitioner.class" property

Question

I am running Cloudera provided Kafka version 0.9.0 and I have written a custom Producer

I am adding the following properties to my ProducerConfig:

        Properties props = new Properties();
        props.put("bootstrap.servers", brokers);
        props.put("acks", "all"); 
        //props.put("metadata.broker.list", this.getBrokers());
        //props.put("serializer.class", "kafka.serializer.StringEncoder");
        props.put("retries", 0);
        props.put("batch.size", 16384);
        props.put("linger.ms", 1);
        props.put("buffer.memory", 33554432);
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("partitioner.class", "com.vikas.MyPartitioner");

However when I am running my program, I am getting this error

16/08/23 18:00:33 INFO producer.ProducerConfig: ProducerConfig values:
        value.serializer = class org.apache.kafka.common.serialization.StringSerializer
        key.serializer = class org.apache.kafka.common.serialization.StringSerializer
        block.on.buffer.full = true
        retry.backoff.ms = 100
        buffer.memory = 33554432
        batch.size = 16384
        metrics.sample.window.ms = 30000
        metadata.max.age.ms = 300000
        receive.buffer.bytes = 32768
        timeout.ms = 30000
        max.in.flight.requests.per.connection = 5
        bootstrap.servers = [server01:9092, server02:9092]
        metric.reporters = []
        client.id =
        compression.type = none
        retries = 0
        max.request.size = 1048576
        send.buffer.bytes = 131072
        acks = all
        reconnect.backoff.ms = 10
        linger.ms = 1
        metrics.num.samples = 2
        metadata.fetch.timeout.ms = 60000

16/08/23 18:00:33 WARN producer.ProducerConfig: The configuration partitioner.class = null was supplied but isn't a known config.

As per the documentation on http://kafka.apache.org/090/documentation.html#producerconfigs

partitioner,class is a valid property but I am not sure why is kafka complaining about it being an unknown config.

Try passing it as a class value, not string: props.put("partitioner.class", MyPartitioner.class) — serejja
@serejja, I tried the change you suggested but Still i am getting the same error — Vikas Saxena
@VikasSaxena it says that the partition.class=null. Are you sure that it passes the class properly? — RadioLog

Sunil Patil Sunil Patil · Accepted Answer · 2016-08-23T14:15:59

Take a look at this article http://www.javaworld.com/article/3066873/big-data/big-data-messaging-with-kafka-part-2.html it has sample on how to use custom partitioner.

As you can see the error message is

"16/08/23 18:00:33 WARN producer.ProducerConfig: The configuration partitioner.class = null was supplied but isn't a known config." which means Kafka does not understand the key partitioner.class

You might want to set partitioner like this

        configProperties.put(ProducerConfig.PARTITIONER_CLASS_CONFIG,"com.vikas.MyPartitioner");

Java Producer not recognizing property "Partitioner.class" property

1 Answers