Native Apache Kafka and Zookeeper with Confluent components?

Question

Could you please tell me about compatibility of Apache Kafka and Zookeeper (native Apache distributuins) with some Confluent's components. I have already installed in my environment Kafka and Zookeeper as a multinodes clusters. But now I need to add schema-registry, kafka-connect.

So I actually tried to deploy Confluent Schema registry from their official docker image. I logged in and was able to successfully telnet kafka broker on port 9093

root@schema-0:/usr/bin# telnet kafka-0.kafka-hs 9093
Trying 10.244.3.47...
Connected to kafka-0.kafka-hs.log-platform.svc.cluster.local.
Escape character is '^]'.

After I tried to do some tests:

# /usr/bin/kafka-avro-console-producer \
  --broker-list localhost:9093 --topic bar \
  --property value.schema='{"type":"record","name":"myrecord","fields" \
  [{"name":"f1","type":"string"}]}'

Add some values:

{"f1": "value1"}

But no luck :(. Got next errors:

value.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer
 (org.apache.kafka.clients.producer.ProducerConfig)

[2018-01-28 11:23:23,561] INFO Kafka version : 1.0.0-cp1 (org.apache.kafka.common.utils.AppInfoParser)

[2018-01-28 11:23:23,561] INFO Kafka commitId : ec61c5e93da662df (org.apache.kafka.common.utils.AppInfoParser){"f1": "value1"}

[2018-01-28 11:23:36,233] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

[2018-01-28 11:23:36,335] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2018-01-28 11:23:36,486] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

Entire system is spinning on Kubernetes

I think you need to set the advertised listener correctly so the producer can reach the broker. — Dmitry Minkovsky
Why not just use Kafka and Zookeeper off of Confluent? They don't modify those sources, only add onto them. In other words, it's fully compatible, but something with your settings (that you've not shown) are incorrect. For example, Kafka default port is 9092 — OneCricketeer

Robin Moffatt Robin Moffatt · Accepted Answer · 2018-01-29T09:27:40

Confluent Platform is Apache Kafka, but with additional components (such as Schema Registry) bundled with it.

The error you're getting is related to the network configuration. You need to make sure that your Broker is available to other nodes, including Schema Registry. In your Schema Registry config you've specified broker-list localhost:9093 but this should be your Kafka broker. In addition as Dmitry Minkovsky mentions, make sure you've set the advertised listener in your broker. This article might help.

Native Apache Kafka and Zookeeper with Confluent components?

1 Answers