Error while using dataflow Kafka to bigquery template

1
votes

I am using dataflow kafka to bigquery template. after launching the dataflow job, it stays in queue for some time then fails with below error:

Error occurred in the launcher container: Template launch failed. See console logs.

When looking at the logs, I see the following stack trace:

at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:192) 
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317) 
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303) 
at com.google.cloud.teleport.v2.templates.KafkaToBigQuery.run(KafkaToBigQuery.java:343) 
at com.google.cloud.teleport.v2.templates.KafkaToBigQuery.main(KafkaToBigQuery.java:222) 
Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata –

While lauching job, i have provided below parameters:

  1. kafka topic name
  2. bootstrap server name
  3. bigquery topic name
  4. SA email
  5. zone.

My kafka topic only contanis message: hello

kafka is installed in gcp instance which is in same zone and subnet as dataflow worker.

2
google-cloud-platformapache-kafkagoogle-cloud-dataflowdataflow
have you tried changing the log settings to show logs at INFO level? The template startup logs are shown as INFO (even the errors). Let me know if that helps. - Pablo
Thanks @Pablo , this is what i can see in INFO section java.lang.RuntimeException: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata - Joseph N
ah I see -- do you see a stack trace around this info log? Usually a stack trace will appear as one log entry per line in the stack - Pablo
@Pablo, at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:192) at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317) at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303) at com.google.cloud.teleport.v2.templates.KafkaToBigQuery.run(KafkaToBigQuery.java:343) at com.google.cloud.teleport.v2.templates.KafkaToBigQuery.main(KafkaToBigQuery.java:222) Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata - Joseph N
@Pablo, just one query.. i have not specified path for UDF . Is it must to provide UDF function path? Basically i am not performing any transformations - Joseph N

2 Answers

5
votes

Adding this here as an answer for posterity:

"Timeout expired while fetching topic metadata" indicates that the the Kafka client is unable to connect to the broker(s) to fetch the metadata. This could be due to various reasons such as the worker VMs unable to talk to the broker (are you talking over public or private ips? Check incoming firewall settings if using public ips). It could also be due to an incorrect port or due to the broker requiring SSL connections. One way to confirm is to install the Kafka client on a GCE VM in the same subnet as Dataflow workers and then verify that the kafka client can connect to the Kafka brokers.

Refer to [1] to configure the ssl settings for the Kafka client (which you can test using the cli on a GCE instance). The team that manages the broker(s) can tell you whether they require SSL connection.

[1] https://docs.confluent.io/platform/current/kafka/authentication_ssl.html#clients

0
votes

Hey Thanks guys for help , i was trying to access kafka with internal ip. it worked when i ched it to public ip. Actually i am running both kafka machines and workers in same subnet. so it should work with internal ip also... i am checking it now