Apache NiFi tuning issues

5
votes

I've developed a NiFi flow prototype for data ingestion in HDFS. Now I would like to improve the overall performances but it seems I cannot really move forward. ​

​The flow takes in input csv files (each row has 80 fields), split them at row level, applies some transformations to the fields (using 4 custom processors executed sequentially), buffers the new rows into csv files, outputs them into HDFS. I've developed the processors in such a way the content of the flow file is accessed only once when each individual record is read and its fields are moved to flowfile attributes. Tests have been performed on a amazon EC2 m4.4xlarge instance (16 cores CPU, 64 GB RAM).

​​This is what I tried so far:

  • ​​Moved the flowfile repository and the content repository on different SSD drives
  • Moved the provenance repository in memory (NiFi could not keep up with the events rate)
  • Configuring the system according to the ​configuration best practices
  • I've tried assigning multiple threads to each of the processors in order to reach different numbers of total threads
  • I've tried increasing the nifi.queue.swap.threshold and setting backpressure to never reach the swap limit
  • Tried different JVM memory settings from 8 up to 32 GB (in combination with the G1GC)
  • I've tried increasing the instance specifications, nothing changes

From the monitoring I've performed it looks like disks are not the bottleneck (they are basically idle a great part of the time, showing the computation is actually being performed in-memory) and the average CPU load is below 60%.

​The most I can get is 215k rows/minute, which is 3,5k rows/second. In terms of volume, it's just 4,7 MB/s. I am aiming to something definitely greater than this. ​ ​Just as a comparison, I created a flow that reads a file, splits it in rows, merges them together in blocks and outputs on disk. Here I get 12k rows/second, or 17 MB/s. Doesn't look surprisingly fast too and let me think that probably I am doing something wrong. ​ ​Does anyone has suggestions about how to improve the performances? How much will I benefit from running NiFi on cluster instead of growing with the instance specs? Thank you all

1
performanceapache-nifi
Would your custom processors be available anywhere to evaluate? In terms of the flow you have established, where does the throughput drop off? Presumably you are using a GetFile followed by your chain listed. - apiri
Yes sure! drive.google.com/drive/folders/… My throughput drops with the custom processors. I expected that since they perform more complex operations than the built-in ones. But still, they just access flowfile attributes and create new ones. Assigning several threads to them seems not to work. However, the point of the last part of the post was that even bypassing them, and just splitting the rows and merging them together I get 12k rows/seconds. Should I move to a higher level and consider batches of rows as work units? Thank you for your time. - riccamini
On average, what is the profile of the flowfiles you are working against in terms of size? One item that jumps out to me is the one processor is toStringing and exerting that pressure on the heap. What version of NiFi are you using? With an m4 class instance, I assume this EBS. What class of EBS is it? GP2? Provisioned iOPS? - apiri
The most I tried the flow with is a 2GB 1,2M rows. Yes, it's a gp2 ebs ssd. I tried evaluating the drive speed using this command as reported on many answers: time sh -c "dd if=/dev/zero of=testfile bs=100k count=1k && sync" 1024+0 records in 1024+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.0617739 s, 1.7 GB/s real 0m0.600s user 0m0.004s sys 0m0.060s - riccamini

1 Answers

5
votes

It turned out the poor performances were a combination of both the custom processors developed, and the merge content built-in processor. The same question mirrored on the hortonworks community forum got interesting feedback.

Regarding the first issue, a suggestion is to add the SupportsBatching annotation to the processors. This allows the processors to batch together several commits, and allows the NiFi user to favor latency or throughput with the processor execution from the configuration menu. Additional info can be found on the documentation here.

The other finding was that the MergeContent built-in processor doesn't seem to have optimal performances itself, therefore if possible one should consider modifying the flow and avoid the merging phase.