Spark Structured streaming ForeachWriter unable to get sparkContext

Question

I'm reading JSON data from Kafka queue usingSpark structured streaming but I need to write the JSON data into Elasticsearch.

However, I can't get sparkContext inside the ForeachWriter to convert the JSON to RDD. It throws NPE.

How can I get SparkContext in the Writer to convert the JSON to RDD?

Answer 1

You cannot. The methods in ForeachWriter run in the executors. You can either write a Elasticsearch sink by yourself, or call Elasticsearch's raw APIs to write the data.