Deleting flowfiles from apache nifi

Question

Below is the flow that I'm working on.

InvokeHTTP-->ExecuteScript-->ExecuteSQL

From invoke HTTP, I'm getting two flow files one with request and another with a response. I've configured ExecuteSQL with Event-Driven scheduling. Since there are two flowfiles in the queue after executing invoke HTTP processor, ExecuteSQL is being triggered two times which is generating duplicate data. I've added ExecuteScript processor with the below groovy code to remove one flowfile from the queue.

import org.apache.nifi.processor.FlowFileFilter;
import org.apache.commons.io.IOUtils

def List<FlowFile> flowFileList = session.get(100)
def size = flowFileList.size();
log.error(size.toString())
int value = size as Integer;
def n=1;

for(FlowFile  k in flowFileList){
if( n!=value ){
session.remove(k)
}
n++;
}

But I'm getting below exception.

org.apache.nifi.processor.exception.FlowFileHandlingException: StandardFlowFileRecord[uuid=27d84996-25a6-41a0-a3e8-06ed2354de18,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1592992841858-2, container=default, section=2], offset=18153, length=500],offset=0,name=72212066-7bcd-456b-ba97-252d18986f72,size=500] transfer relationship not specified

I've also added session.transfer(k,REL_SUCCESS) in for loop. But it's also not working. Can anyone tell me what's wrong here and suggest if there is an easier way of removing all flow files except one to implement the above scenario?

Andy Andy · Accepted Answer · 2020-06-24T15:15:43

It sounds like you have all the relationships from InvokeHTTP going into the next processor. Instead, direct the Original, Failure, Retry, and No Retry relationships to a different destination or Auto-terminate them at the InvokeHTTP processor. Then direct the Response relationship to the ExecuteSQL processor. You will not need the ExecuteScript processor as you will no longer have multiple flowfiles for a single HTTP call.

You should also set the ExecuteSQL processor to be Timer-driven again.

Deleting flowfiles from apache nifi

1 Answers