Connecting to AWS MSK Kafka from my laptop: NoBrokersAvailable

0
votes

I am having trouble connecting to AWS MSK Kafka (with with Access Control Method configured to None) from my laptop, error is: NoBrokersAvailable

I am using plaintext conn over port 9092:

from kafka import KafkaProducer 
kafka_brokers = [ "xxx.kafka.ap-east-1.amazonaws.com:9092","xxx.kafka.ap-east-1.amazonaws.com:9092","xxx.kafka.ap-east-1.amazonaws.com:9092" ]
producer = KafkaProducer(bootstrap_servers=kafka_brokers)

Ref: AWS MSK - plaintext broker port 9092, TLS 9094 https://docs.aws.amazon.com/msk/latest/developerguide/client-access.html

I tried three things (Which did not help):

a. Under my VPC default security group added inbound+outbound source/destination over 9092+9094 source/dest 0.0.0.0/0

b. Tried run over AWS Client VPN following instructions below:

c. Turn off Firewall on my laptop

I see some others asking similar questions: How to connect Python consumer to AWS MSK

Some folks suggesting it's simply not-supported for clients on local machines/laptop to connect to AWS MSK: How can I access my AWS MSK managed kafka queue from my local machine and EC2 instances in other regions Is this true/official?

I'm thinking create EIP (Elastic IP) https://repetitive.it/aws-msk-how-to-expose-the-cluster-on-the-public-network/?lang=en Then somehow associate with broker domain names xxx.kafka.ap-east-1.amazonaws.com (but not sure how) and configure inbound outbound rules.

But I haven't been able to find anything that helps.

I'm on my wits end, this problem already consumed whole day. Any help greatly appreciated.

Thanks

1
python-3.xamazon-web-servicesapache-kafkaaws-msk
Have you checked if the security groups are setup correctly and allowing traffic to your laptop. Also can you confirm if the brokers are in public or private subnet? – floating_hammer
They are in Public subnet. And I configured inbound + outbound rules to allow 9092/9094, no luck. – user3761555
I would suggest create an ec2 instance and then try connecting using that to your msk cluster. – floating_hammer
Last I checked, if the advertised listeners (and firewall) is correctly configured, you can connect to any broker from any network... If you use kafkacat -L on your brokers, what addresses do they give? Can you telnet those addresses from outside of AWS? – OneCricketeer

1 Answers

0
votes

It won't be possible to directly connect to MSK brokers from you laptop . Although MSK listeners exposes a public DNS name, it actually resolves to a private IP of the MSK broker. So since you are not within the VPC, you cannot reach the private IP.

The cheapest option to achieve this is to setup
AWS Client VPN endpoint.

(edited) Observed you have already tried the client VPN, what did not work? I am using it.