How to programmatically create a Topic in Kafka through spark structured streaming

0
votes

I want to create multiple kafka Topics run time in my Spark Structured Streaming application. I found that there are various methods available in Java API. But I couldn't find any with Spark Structured Streaming. Please let me know if there is any way available or I need to use java library

My apache Spark version is 2.4.4 and Kafka library dependency is spark-sql-kafka-0-10_2.12

1
apache-sparkapache-kafkaspark-structured-streaming
if the topic does not exist, Spark create it automatically – Ahmed
Thanks, I really don't know – Dhara Dhruve
@Ahmed Does the broker need to have auto topic creation enabled? If not, how would Spark create a topic without setting replication or partition settings? – OneCricketeer
With auto creation off, I'd expect an exception. Same without proper credentials when using a secure Kafka cluster. – Jacek Laskowski

1 Answers

-1
votes

AFAIK, Spark doesn't create topics.

You can use the same Java APIs you've found before initializing your SparkSession

spark-sql-kafka includes kafka-clients, so you have the AdminClient class available

How to create a Topic in Kafka through Java