How to register custom Spark structured streaming source

Question

I need to create a custom streaming source by extending FileStreamSource. The idea is to override commit, so that processed files (S3 objects in this case) are renamed to have a certain prefix. However, I don't know how to use this custom source. Obviously I don't want to compile Spark -- the application will be running on Amazon EMR clusters.

bp2010 bp2010 · Accepted Answer · 2018-07-13T07:50:05

Once you create your custom source in your project, you need to register it in the format of a DataStreamReader:

val input = spark
  .readStream
  .format("path.to.MyCustomFileStreamSource")
  .load

How to register custom Spark structured streaming source

1 Answers