how do i add a topic to a running kafka container using docker commands?

Question

I have a kafka container started using following

docker run --detach --name kafka -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=192.168.1.89 --env ADVERVTISED_PORT=9092 --env AUTO.CREATE.TOPICS.ENABLE spotify/kafka

i can use docker logs kafka to see its started.

I then created a simple groovy script client producer to write some entries, however this keeps erroring with

> Sending metadata request {topics=[wills topic]} to node 0
> Error while fetching metadata with correlation id 1 : {wills topic=INVALID_TOPIC_EXCEPTION}
....

I have set the following properties in the client code

    Properties props = new Properties()
    props.put("bootstrap.servers", "192.168.1.89:9092" )   //Assign localhost id and external port (9092 int)
    props.put("acks", "all")                            //Set acknowledgements for producer requests.
    props.put("retries", 0)                             //If the request fails, the producer can automatically retry,
    props.put("batch.size", 16384)                      //Specify buffer size in config
    props.put("linger.ms", 1)                           //Reduce the no of requests less than 0
    props.put("buffer.memory", 33554432)                //The buffer.memory controls the total amount of memory available to the producer for buffering.
    props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer")
    props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer")
    props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")
    props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")
    props.put ("auto.create.topics.enable", true)       //enable auto topic creation

    producer = new org.apache.kafka.clients.producer.KafkaProducer<String, String>(props)

    for(int i = 0; i < 10; i++) {
        producer.send(new ProducerRecord<String, String>(topicName, Integer.toString(i), Integer.toString(i)))
    }
    println("Message sent successfully")
    producer.close()

but that does not not seem to enable auto topic creation. Given that i've spun the kafka server up as docker container - how does one administer that container instance once it is up and running using docker commands as the means to talk to the container

there is a note on the kafka site along the line of running a shell commend "bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test"

but its not easy to see how one edits/interacts with this in container once its up.

how does one administer the kafka container instance in docker once it has been started. otherwise what to you have to add to the docker run command to get topics pre created at startup

Advice greatfully received at this point

Sebastian Sebastian · Accepted Answer · 2017-01-29T15:45:56

The error message {wills topic=INVALID_TOPIC_EXCEPTION} is indicating that you are using wills topic as topic name.

A topic name cannot contain whitespaces. Try to rename to wills_topic and it should solve the problem.

This regex describes the legal characters for topic names (check sources):

val legalChars = "[a-zA-Z0-9\\._\\-]"

Use docker exec -it <container-name> <command> to launch the Kafka admin tools. Or just open a bash within your container docker exec -it <container-name> bash (see Docker docs).

how do i add a topic to a running kafka container using docker commands?

2 Answers