Inserting from Kafka to Cassandra using NIFI

Question

I have a requirement to read huge CSV file from Kafka topic to Cassandra. I configured Apache Nifi to achieve the same.

Flow:

User does not have a control on Nifi setup. He only specifies the URL where the CSV is located. The web application writes the URL into kafka topic. Nifi fetches the file and inserts into Cassandra.

How will I know that Nifi has inserted all the rows from the CSV file into Cassandra? I need to let the user know that inserting is done.

Any help would be appreciated.

Answer 1

I found the solution.

Using MergeContent processor, all FlowFiles with the same value for "fragment.identifier" will be grouped together. Once MergeContent has defragmented them, we can notify the user.