Use Case: An application uses spark to process data for 5 minutes, the data to be processed could be of several hundred thousands of records in data storage. The choice for data storage is Elastic Search.
Issue: Do we have a connector for the spark in elasticsearch similar to the connector in MongoDB?
https://www.mongodb.com/products/spark-connector.
Investigation: I spent a lot of time but the best I could find was a solution using search API with scroll(we can fetch the limited number of records for given number interval), but this does not fit my use-case.
Please note that my elastic search will have JSON data and we do not want to save RDD. as mentioned in below
https://www.elastic.co/guide/en/elasticsearch/hadoop/master/spark.html
