Unable to connect to Kafka Broker on Hortonworks Sandbox (HDP 2.6) from my Windows Host Machine on Port 6667

1
votes

I have downloaded Hortonworks Sandbox (IP: 192.179.144.XXX) on my Windows 10 laptop. I have the HDP Sandbox running on VMWare.

Below is my sandbox information: 

Created on: 19_04_2017_19_09_16 for 
Hadoop stack version:  Hadoop 2.7.3.2.6.0.3-8 
Ambari Version: 2.5.0.5-1 
Ambari Hash: 0b5e975972e7a0b265e87b2e38eefde9039ef44c 
Ambari build:  Release : 1 
Java version:  1.8.0_121 
OS Version:  CentOS release 6.9 (Final)</code>

My goal is to be able to connect to Kafka (HDP Sandbox) from Java IntelliJ SDK based out of my Windows host machine.

So far, I have tried the following in order to be able to access HDP Kafka from my host machine via Java and/or Kafka tool 1.0, but been unsuccessful.

  1. Updated the Network-Atapter to 'Host Only' in my VMware settings.

  2. Opened, Kafka broker, port number 6667 by adding the port number to the 'start_sandbox.sh' file. Leveraged the following article: http://tlxu.blogspot.com/2016/12/add-ports-mapping-to-hdp-25-vmware.html

    2.1 Below are the steps taken to "open" the port 6667 - 2.1.1) login to the Sandbox VM (actual docker container) 2.1.2) Disable sandbox.service: $ systemctl disable sandbox.service 2.1.3) Reboot the VM: $ init 6 2.1.4) Modify sandbox start script: $ vi /root/start_scripts/start_sandbox.sh 2.1.5) Added "-p 6667:6667 \" after "-p 2222:22 \" 2.1.6) save and exit. 2.1.7) Delete existing sandbox container: $ docker rm sandbox 2.1.8) Enable sandbox.service: systemctl enable sandbox.service 2.1.9) Reboot the VM: $ init 6 2.1.10) Verify new ports: $ docker ps | grep 6667 2.1.11) Received the following output - 0.0.0.0:6667->6667/tcp

  3. However, when I open chrome on my laptop and try to connect to: http://192.179.144.XXX:6667, I receive the following error:

    This site can’t be reached The webpage at http://192.179.144.XXX:6667/ might be temporarily down or it may have moved permanently to a web address. ERR_UNSAFE_PORT

  4. I also tried 'logging' and 'telnetting' on the sandbox using "Putty" with following credentials: [email protected] and port: 6667. I receive the following error:

    Network Error: software caused connection abort

  5. Within the sandbox I ran the following command to check if the port is open inside the sandbox. I don't know what the output means, though.

    [root@sandbox ~]# netstat -tnlpa | grep 6667 [root@sandbox ~]#

  6. Kafka properties in "Kafka Broker" section in Ambari has the following properties:

    Kafka broker host: sandbox.hortonworks.com zookeeper.connect: sandbox.hortonworks.com:2181 listeners: PLAINTEXT://localhost:6667

    (Please note: I also tried changing 'localhost' to '0.0.0.0', and my kafka-console-consumer and kafka-console-produer failed. Received "LEADER_NOT_AVAILABLE" error.)

  7. Within Kafka properties in Ambari, I logged in as an Admin and added the following properties to the "Custom Kafka-broker"

    advertised.port=6667 advertised.listeners=PLAINTEXT://sandbox.hortonworks.com:6667 advertised.host.name=192.179.144.XXX

I am unable to move forward, and I would really appreciate if anyone could help out on:

1.) Is my port open? If not, how else can I open the port?
2.) Any additional changes I need to make with Kafka settings, that would let the Java IDE on my Windows to connect to Kafka (dependent on issue #1)?

1
javaapache-kafkaporthortonworks-sandbox
No, the port is closed. You will have to look at the logs to find out why. Try systemctl status sandbox.service. – Robin Green

1 Answers

0
votes

Kafka is not an HTTP service. Chrome, or browser, will not connect.

Kafka is not an SSH program. Putty, or other, will be connect.

You need to verify connection using kafka-console-* tools from windows 

listeners: PLAINTEXT://localhost:6667

Should be the following for Kafka to listen externally of the sandbox 

 listeners: PLAINTEXT://:6667

Then, I don't know about Host Only or NAT settings, but you shouldn't need to touch those if you setup port forwarding in the VM settings. 

advertised.host.name

Should not need to be changed in Ambari, but if you did, then set it to the sandbox hostname

If you don't need Hadoop and all the other stuff in HDP, I suggest getting the HDF sandbox, or just using Docker for Windows and using the Kafka containers on DockerHub