NiFi: Routing flowfiles evenly to different processors

1
votes

I am trying to use NiFi's PutCassandraRecord to input data into a Cassandra cluster of 3 nodes(provided all 3 nodes address to PutCassandraRecord). The throughput I am getting isn't enough for my workload, so I tested using 3 different PutCassandraRecord processors and providing 1 node address to each processor. I was able to get greater throughput this route.

My question is could I split the amount of incoming flowfiles evenly to 3 or more different processors? More specifically, I am using ListenHTTP to listen for incoming data, and I want to split the incoming data evenly to 3 different PutCassandraRecord processors

I am running NiFi 1.9.2

1
cassandraapache-nifi

1 Answers

2
votes

You could use the DistributeLoad processor with round-robin distribution.

Also, you could try having a single PutCassandraRecord processor, but increase the concurrent tasks of the processor from 1 to 3. This should be similar to having 3 separate processors, but I don't know anything about Cassandra so I don't know the difference between specifying all nodes vs only one.