Apache Nifi - Split a large Json file into multiple files with a specified number of records

Question

I am a newbie to Nifi and would like some guidance please.  We want to split a large Json file into multiple files with a specified number of records. I am able to split a file into individual records using SplitJson and the Json Path Expression set as $..* I have also added an UpdateAttribute Processor with filename set to ${filename}_${fragment.index} so that we have the sequence of the files as order is important. However, we might want to have say a 100,000 records split into 100 files of 1000 records each . What is the easiest way to do this ? Thanks very much in advance

Answer 1

There is a SplitRecord processor. You can define the number of records to be split per file, such as:

Record Reader      CSVReader
Record Writer      CSVRecordSetWriter
Records Per Split  3

I have tested with the record,

id
1
...
8

and it is split into 3 files with the id = (1,2,3), (4,5,6), (7,8).