Using Kinesis producer library in Spark

Question

I have Spark job reads millions of records from Cassandra, filter out(business rules) and write to Kinesis stream. I don't find any example and testimonial on how to invoke KPL(Kinesis Producer Library) from Spark. Is that correct approach? Do I have any other option?

Answer 1

you can create KPL producer per partition and then for each partition you can send the message. Keep the partitions small to avoid overloading task/core nodes.