Kafka Topic Partitioner function - configuration question

Question

I am trying to write a Kafka producer using librdkafka (c code library). I do not want my application to have to manually round-robin through the number of partitions on a topic. In the Java version of the Kafka Producer, this is possible by not specifying the partition or key. The documentation in librdkafka states the following:

 * \p partition is the target partition, either:
 *   - RD_KAFKA_PARTITION_UA (unassigned) for
 *     automatic partitioning using the topic's partitioner function, or
 *   - a fixed partition (0..N)
RD_EXPORT
int rd_kafka_produce(rd_kafka_topic_t *rkt, int32_t partition,
              int msgflags,
              void *payload, size_t len,
              const void *key, size_t keylen,
              void *msg_opaque);

So what is the topic's partitioner function? I cannot find documentation on this anywhere. Is this round-robin by default? Is this something I need to enable in Zookeeper?

Published in librdkafka is defined as: int rd_kafka_produce(rd_kafka_topic_t *rkt, int32_t partition, int msgflags, void *payload, size_t len, const void *key, size_t keylen, void *msg_opaque); — cman227
Just curious: What do you want to partition by? Notice in the produce method, you can manually assign a partition, so you don't need a separate partition function — OneCricketeer
I would like round robin, but I don’t want the publisher to have to know the number of partitions. — cman227
Automatic partitioning would be calculating the hash, then mod-ing by the partition count, so it's not exposed to the producer anyway — OneCricketeer

Mickael Maison Mickael Maison · Accepted Answer · 2018-10-26T15:17:54

If you don't explicitly specify the partition when producing, by using RD_KAFKA_PARTITION_UA, librdkafka will automatically determine the partition using a partitioner. The strategy is defined by a setting named partitioner, the default is called consistent_random and uses

CRC32 hash of key (Empty and NULL keys are randomly partitioned)

See https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md for the full details. Also note that if you want exactly the same behaviors as Java you need to set partitioner to murmur2_random.

Finally you can also write your own partitioner logic. In that case, you need to set partitioner_cb or rd_kafka_topic_conf_set_partitioner_cb()

Kafka Topic Partitioner function - configuration question

1 Answers