Invoke API from NIFI

0
votes

I have a requirement where I need to invoke API at the end of successful data dump to cassandra table. I have used below processors in my nifi flow.

QueryDataBaseTableRecord --> SplitLines --> ExecuteScript --> PutCassandraQL --> InvokeHTTP --> Invoke HTTP --> PutS3

Usecase: Need to dump the data from DB2 to Cassandra, Once after the successful dump it should invoke first API InvokeHTTP(this API has to dump around 100k records to cassandra and send the response message as "Data Inserted" back to nifi) , once we get this response message the second API needs to be invoked InvokeHTTP.

Problem is Nifi automatically routes to failure flow (after ~1 minute or so) as its not waiting for response message from 1st Invoke API, but when we checked at the backend cassandra end data dumping is still in progress.

Could anyone please let me know - How to make invokehttp processor to wait for API response message and route to next step ? Appreciate your suggestion if there is any other effective/alternative way we can try to handle this usecase.

1
apache-nifi
check your timeout setting of the InvokeHTTP processor. - Lamanus
Increase the value for Connection Timeout and Read Timeout. the default value is 5 and 10. Increase and check based on the fact. - natarajan k
Yes I tried increasing both the timeouts, still it routed to failure mode its not waiting for response message. Observed that if the API has very less number of records (less than 1000's) am getting proper response message through nifi. - iyasac

1 Answers

0
votes

If you check the configuration ot InvokeHTTP you have two timeout fields:

  1. Connection Timeout
  2. Read Timeout

Just increase the value for those.

Here is the documentation