Connect Confluent with already existing three kafka brokers

4
votes

I'm new in Confluent world, and I know how to start kafka, zookeepers from confluent, but it's not that what I need.

I have already 3 kafka nodes and 2 zookeepers installed by Ambari. Afterwards I downloaded 3.0.0 version of Confluent and now I want to connect Confluent with already running Kafka and zookeeper. I don't want to instance new kafka server or zookeeper server which confluent is giving.

Does anyone has an idea how to accomplish that, what to actually run from Confluent and what to change.

By now I was only chaning files in ./etc/kafka or ./etc/zookeeper which are in Confluent dir. Thank you!

1
apache-kafkaapache-zookeeperconfluent-platform
Your question is unclear to me. You should check out Confluent webpage and docs first: confluent.io/product and docs.confluent.io/3.0.0 Confluent's shipped Kafka and ZK version are the exact same as Apache Kafka 0.10.0 (for CP 3.0.0). – Matthias J. Sax
I have some issuses with understanding how Confluent is working. Let's suppose I have four servers in cluster. I already have installed Kafka ond the server1-3, and zookeepers on server1-2. On server4 I have installed confluent. Is it possible to connect Confluent with my Kafka brokers on server1-3 or he is working on different way? Do I need to have confluent on all four of my or three server to have confluent+kafka+zk integration? Can you clear that maybe for me? – iMajna
The CP stack contains multiple components. So which one do you want to use? Schema Registry? Rest Proxy? You should be able to use all of those in combination with you running Kafka deployment. Please read the CP docs. You question is too broad to give an helpful answer. – Matthias J. Sax
Lets suppose that I don't know anything. What do you propose me how tu run Confluent on multi node cluster? After that, can I run Confluent AND Kafka ( but Kafka as already installed service, not the one that Confluent gives) alongside connected? Thank you :) – iMajna
I need to flag this question as "unclear what you are asking". Sorry. You need to be more specific. This is too broad. SO in not for providing tutorials. I can just tell you, that it should be possible to connect Confluent Platform to you running Kafka installation. – Matthias J. Sax

1 Answers

1
votes

clarify some basics about Confluent and how manage communication between Confluent and Kafka

First things first, there is no single application called "Confluent" that can be started all on its own.

There's is nothing to configure for Kafka or Zookeeper. The Confluent Platform doesn't add anything on top of the existing Apache Kafka you have (presumably, via Hortonworks or Cloudera).

In fact, those companies do add patches to Kafka that would be slightly different than the base Apache versions you would get from Confluent.

That being said, if you read through each of extra services that Confluent provides, you'll notice either a Zookeeper or a Bootstrap server configuration option. Fill out those fields, start the respective services, and you're good to go.

what to actually run from Confluent

Look in the bin directory, you can find all the start scripts. From the comments, looks like you're trying to use Connect Distributed (which should already be installed by any recent Kafka installation, it's not Confluent specific), and Schema Registry. You'll have to be more specific about the errors that you get, but the config files are all in the etc path.

Unless you're using KSQL, REST Proxy or Control Center, there's not much to run because, as mentioned, Kafka Connect is included with the base Apache Kafka project and Hortonworks is maintaining their own Schema Registry project

2 zookeepers installed by Ambari

This is a highly non-recommended setup. Please install an odd number of Zookeepers. 3 or 5, preferably