java.lang.RuntimeException: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata

0
votes

I am using dataflow kafka to bigquery template. after launching the dataflow job, it stays in queue for some time then fails with below error:

java.lang.RuntimeException: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata

While lauching job, i have provided below parameters:

  1. kafka topic name
  2. bootstrap server ip and port (private ip)
  3. bigquery topic name
  4. SA email
  5. zone.
  6. Network , subnet (same network where kafka server running)

My kafka topic only contanis message: hello

kafka is installed in gcp instance which is in same zone and subnet as dataflow worker.

1
apache-kafkagoogle-cloud-dataflowdataflow
The issue seems to be on the side of Kafka, a possible solution is mentioned in the following Stackoverflow thread where is mentioned that the issue can happen when trying to connect to a broker expecting SSL connections and the client config did not specify and the solution is to set the security protocol to SSL: security.protocol=SSL - Messier_31
Make sure your Kafka broker listeners properties are setup to bind externally - OneCricketeer
Hey Thanks @OneCricketeer , i was trying to access kafka with internal ip. it worked when i ched it to public ip. Actually i am running both kafka machines and workers in same subnet. so it should work with internal ip also... i am checking it now - Joseph N

1 Answers

1
votes

This issue is solved. i was trying to access kafka bootstrap server with internal ip. it worked when i changed it to public ip.