Create Kafka topic with N partitions

Question

I'm using Confluent Kafka DotNet lib to create and produce to topics:

producer.ProduceAsync(topic, key, message);

When this topic doesn't exist, the topic is automatically created with one partition.

But for testing purposes, I want to be able to create a topic with N partitions programmatically.

I'm unable to find any examples here (branch 0.11.5), how can I create a topic with multiple partitions? https://github.com/confluentinc/confluent-kafka-dotnet

github.com/confluentinc/confluent-kafka-dotnet/blob/master/src/… should be useful — Giorgos Myrianthous
That's unreleased code right now, but yes could be a good option if needed. — FBryant87
Without using the AdminClient, you're only other options to 1) building the master branch or 2) shelling out to the Zookeeper CLI commands would be figure out the binary topic create protocol request on your own — OneCricketeer

FBryant87 FBryant87 · Accepted Answer · 2018-10-09T16:18:29

Since I was already using Docker I found out it was easiest to simply create a test topic (with N partitions) in the docker-compose:

  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"

  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: kafka
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_CREATE_TOPICS: "PartitionsTest:3:1"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    links:
      - zookeeper

Which creates topic "PartitionsTest" with 3 partitions.

Warning: Your tests may execute before the topic has time to create itself.

Once the Admin feature is available in a release version of the Confluent Kafka library, I'll probably update it to use that approach.

Create Kafka topic with N partitions

1 Answers