Kafka Connect - Could not look up partition metadata

Question

Running Kafka Connect on a Kafka cluster which has auto creation of topics turned off. As far as I could see on another cluster that is working we had the topics connect-configs, connect-offsets, and connect-status created. So these topics were recreated on the new cluster with same replication factor and partitions.

We are getting an error when starting the service though I have netcat from connect instance to brokers and there was no connection issue there

Could not look up partition metadata for offset backing store topic in allotted period. This could indicate a connectivity issue, unavailable topic partitions, or if this is your first use of the topic it may have taken too long to create

I can see where this is thrown below from the connect utils class. But have checked and these topics are created so a bit lost now

        producer = createProducer();
        consumer = createConsumer();

        List<TopicPartition> partitions = new ArrayList<>();

        // We expect that the topics will have been created either manually by the user or automatically by the herder
        List<PartitionInfo> partitionInfos = null;
        long started = time.milliseconds();
        while (partitionInfos == null && time.milliseconds() - started < CREATE_TOPIC_TIMEOUT_MS) {
            partitionInfos = consumer.partitionsFor(topic);
            Utils.sleep(Math.min(time.milliseconds() - started, 1000));
        }
        if (partitionInfos == null)
            throw new ConnectException("Could not look up partition metadata for offset backing store topic in" +
                    " allotted period. This could indicate a connectivity issue, unavailable topic partitions, or if" +
                    " this is your first use of the topic it may have taken too long to create

@cricket_007 I will tomorrow when back at laptop, but from looking at documentation I am thinking the problem may be the offsets storage topic. I know it wasn't created with a compacted cleanup policy. Not sure this would matter though? — AnonymousAlias
It wasn't though because the topics were created manually, this solved my issue. Post it as an answer if you want and I will accept — AnonymousAlias

OneCricketeer OneCricketeer · Accepted Answer · 2020-02-19T14:07:34

the topics were created manually,

Kafka Connect will use the AdminClient to initialize its own topics. There's no reason to do it manually.

If you do, you need to ensure the correct settings, which are documented in both Confluent and Apache Kafka documentation for getting started with Connect

Kafka Connect - Could not look up partition metadata

1 Answers