Kafka 8.2.2 Dynamic Topic drops first event

Question

EDIT : I am seeing the same exact behavior with the Kafka 9 Consumer API also.

I have a simple Kafaka 8.2.2 Producer with the enable topic creation property set to true. It will create a new topic when an event with a non-existent topic is created, but the event that creates that topic does not end up in Kafka and the RecordMetadata returned has no errors.

    public void receiveEvent(@RequestBody EventWrapper events) throws InterruptedException, ExecutionException, TimeoutException {
    log.info("Sending " + events.getEvents().size() + " Events ");
    for (Event event : events.getEvents()) {
        log.info("Sending Event - " + event);
        ProducerRecord<String, String> record = new ProducerRecord<>(event.getTopic(), event.getData());

        Future<RecordMetadata> ack = eventProducer.send(record);
        log.info("ACK - " + ack.get());
    }

    log.info("SENT!");
}

I have a program that polls for new topics (I wasn't happy with the dynamic/regex topic code in Kafka 8) and it finds the new queue and subscribes, and it does see subsequent events, but never that first event.

I also tried the kafka-console-consumer script and it sees that exact same. First event never seen, then after that events start flowing.

Ideas?

Gandalf Gandalf · Accepted Answer · 2016-02-19T00:22:51

Turns out there is a property you can set props.put("auto.offset.reset","earliest");

And after setting this, the Consumer does receive the first event put on the topic.

Kafka 8.2.2 Dynamic Topic drops first event

1 Answers