Writing dataframe to kafka topic in an avro format for spark < 2.4?

0
votes

Q1. Considering I have a dataframe df and a schema myschema, how do I proceed to write the dataframe into kafka topic in an avro format ?

Q2. Is there any optimized way if we do not consider udf ?

Most of the available solutions are for spark > 2.4 where they have inbuilt avro functions to use.

1
apache-sparkapache-kafkaavrospark-avro
Does this answer your question? Spark Dataframe write to kafka topic in avro format? – Giorgos Myrianthous
I tried, from there, eventDF.select( encodeUDF(struct(eventDF.columns.map(column):_*)).alias("value") ) struct and column was showing red in color, could you please help me in defining these select query – supernatural
1) Are you using the Schema Registry? 2) They are read because you never defined/imported them – OneCricketeer
This docs page is correct, if you are not using Schema Registry spark.apache.org/docs/latest/… – OneCricketeer

1 Answers

0
votes

Most of the available solutions are for spark > 2.4 where they have inbuilt avro functions

That inbuilt function was an external library, but was later merged in to the main Spark project. If you have < 2.4, I suggest you upgrade you ultimately upgrade your Spark cluster, or refer the docs there.