Camel-Kafka Integration Issue

Question

I am trying Camel-Kafka integration.
I have two queues :
queue1 and queue2.

There are three routes :

1. Route1 puts a list of two messages in queue1 (It should do it only once).
2. Route2 reads the list from queue1, splits it, and puts the individual messages in queue2
3. Route3 reads the messages from queue2 and just prints it.

The code is as follows :

import java.util.ArrayList;
import java.util.List;

import org.apache.camel.CamelContext;
import org.apache.camel.Exchange;
import org.apache.camel.Processor;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.impl.DefaultCamelContext;

public class CamelListTest {
    public static void main(String[] args) throws Exception {
        CamelContext context = new DefaultCamelContext();
        context.addRoutes(new CamelListRoute());
        context.start();
        Thread.sleep(30000);
        context.stop();
    }
}

class CamelListRoute extends RouteBuilder {
    @Override
    public void configure() throws Exception {


        //Route1, expected to run once
        from("timer://timerName?repeatCount=1").process(new Processor() {
            @Override
            public void process(Exchange exchange) throws Exception {
                List<String> inOrderList = new ArrayList<String>();
                inOrderList.add("1");
                inOrderList.add("2");
                exchange.getIn().setBody(inOrderList, ArrayList.class);
            }
        })
        .to("kafka:<ip>:9092?topic=queue1");


        //Route2
        from("kafka:<ip>:9092?topic=queue1&groupId=testing&autoOffsetReset=latest&consumersCount=1")
        .split()
        .body().process(new Processor() {
            @Override
            public void process(Exchange exchange) throws Exception {
                System.out.println("2nd Route : " + (exchange.getIn().getBody().toString()));
            }
        })
        .to("kafka:<ip>:9092?topic=queue2");


        //Route3
        from("kafka:<ip>:9092?topic=queue2&groupId=testing&autoOffsetReset=latest&consumersCount=1")
        .process(new Processor() {
            @Override
            public void process(Exchange exchange) throws Exception {
                System.out.println("3rd Route : " + (exchange.getIn().getBody().toString()));
            }
        });
    }
}

It is not working as expected, and there are few issues observed :

1. The first route, which is expected to run only once (repeatCount=1), runs continuously, putting the same message in queue1 again and again.
2. The second route reads the messages from queue1, splits it, but does not put it in queue2
3. Since second route does not put anything in queue2, this route does not get any messages.

Can anyone help me figure out what is wrong here?

Answer 1

I see couple of things:

1. I hope you are giving Kafka Url like this: "kafka://localhost:9092?topic=queue1"

note: kafka://

2. Providing zookeeper urls for consumers eg: kafka://?topic=queue1&zookeeperConnect=&consumerStreams=1&groupId=testing&autoOffsetReset=largest

3. Note in previous point autoOffsetReset value will be largest or smallest instead of latest.

Answer 2

I think you shoud exchange the message.

in processor do something like:

exchng.getOut().setHeader("type", "queue"); exchng.getOut().setBody(exchng.getIn().getBody() );

then could add a choice in the second route, does not require the third route.

Answer 3

I believe the first issue of running continuously and putting the same message in queue1 again and again is happening because you are using the same consumer groupId, groupId=testing for both your kafka consumers in routes 2 and 3.

Amend the kafka consumers to consume from different groupIds like so and this will ensure that the message is not consumed again and again.

//Route2
from("kafka:<ip>:9092?topic=queue1&groupId=testing-queue1&autoOffsetReset=latest&consumersCount=1")

and

//Route3
from("kafka:<ip>:9092?topic=queue2&groupId=testing-queue2&autoOffsetReset=latest&consumersCount=1")

The other issues of producing to queue2 and consuming from it to print, I think might be due to version incompatibilities. I have used camel-kafka version 2.20.1 (that uses kafka-clients 0.11.0.1 under the hood) and 2.21.0 (that uses kafka-clients 1.0.0 under the hood) and changed the routes to reflect the changes like so and this seems to consume-produce-consume fine.

class CamelListRoute extends RouteBuilder {
    @Override
    public void configure() throws Exception {


        //Route1, expected to run once
        from("timer://timerName?repeatCount=1").process(new Processor() {
            @Override
            public void process(Exchange exchange) throws Exception {
                List<String> inOrderList = new ArrayList<String>();
                inOrderList.add("1");
                inOrderList.add("2");
                exchange.getIn().setBody(inOrderList, ArrayList.class);
            }
        })
        .to("kafka:queue1?brokers=<ip>:9092");


        //Route2
        from("kafka:queue1?brokers=<ip>:9092&groupId=testing-queue1&autoOffsetReset=latest&consumersCount=1")
        .split()
        .body().process(new Processor() {
            @Override
            public void process(Exchange exchange) throws Exception {
                System.out.println("2nd Route : " + (exchange.getIn().getBody().toString()));
            }
        })
        .to("kafka:queue2?brokers=<ip>:9092");


        //Route3
        from("kafka:queue2?brokers=<ip>:9092&groupId=testing-queue2&autoOffsetReset=latest&consumersCount=1")
        .process(new Processor() {
            @Override
            public void process(Exchange exchange) throws Exception {
                System.out.println("3rd Route : " + (exchange.getIn().getBody().toString()));
            }
        });
    }
}