Queue monitoring tool in NiFi using OAuth and Keycloak

Question

I want to be made aware of when the overall queue size in NiFi exceeds a certain value. I have tried to use the NiFi toolkit, but I've been unable to find anything in there that represents the total queue size.

I have been attempting this, which should allow me to access the connections in a group of processes in NiFi through the API. However, I cannot connect successfully to the API from NiFi.

I get the following error in NiFi when trying to connect to the API without credentials.

Unknown user with identity 'anonymous'. Contact the system administrator.

I believe I'll be able to view the queue size through the REST API, however in order to do this I'll need to authenticate using OAuth and Keycloak, and I'm not sure if this is possible since 2FA is required.

Any assistance in this issue would be much appreciated.

if you are able to see the required information in nifi web interface then you should be able to get it through nifi web-api. all authentication issues you should solve with keycloak admin. it's possible to simplify authentication process for system-to-system integration in keycloak. — daggett

Fabio Torchetti Fabio Torchetti · Accepted Answer · 2020-10-26T10:57:14

Remember that you can always use certification based authentication to interact with the APIs. Create a certificate using the same CA you used to authenticate between nodes in the NiFi cluster, add the user to NiFi and then use the private key to make the request.

For example, using curl, you could hit the endpoint like this:

curl --cacert nifi-ca.crt \
  --key api-client.key --cert api-client.crt \
  https://nifi-001:8443/flowfile-queues/{id}/flowfiles/{flowfile-uuid}/content

This assumes you have configured HTTPS on your nodes.

Queue monitoring tool in NiFi using OAuth and Keycloak

1 Answers