Create Nifi PutSQL Processor with Python

Question

I want to transfer my Nifi-ETL pipeline (mainly PUTSQL-processors) from my development instance to my production instance of Apache Nifi, optimally with Python for re-usability.

I thought I would give it a shot to just try and copy-paste them.

GET processor from DEV with GET-request on /nifi-api/processors/{id}
PUT processor to PRD Nifi-instance with PUT-request on /nifi-api/processors/{id}

Code:

# GET processor and parse to JSON
response = requests.get(nifi_url_dev + 'processors/' + proc_id
                            , headers=header)
processor = json.loads(response.content)


# PUT processor 
processor['revision']['version'] = 0 # reset version
payload = json.dumps(processor).encode('utf8')
response = requests.put(nifi_url_prd + 'processors/' + proc_id
                        , data=payload
                        , headers=header)

This failed on the PUT with a 409 HTTP Conflict Error. I am guessing this is because I am trying to put a ressource on an URI that expects a resource to exist already at that place.

The documentation lists "Create a processor, Set properties, Schedule" next to the processor APIs, but when looking into it, there is no dedicated API for creation - I decided to go with PUT because it says "Updates a processor" which is the closest thing I can see in there to creating a new one from scratch.

Do you have any ideas on how to create processors with Python? Either by copying existing ones or creating entirely new ones?

If you're going to be using the API & Python I'd suggest using Nipyapi github.com/Chaffelson/nipyapi - but it sounds like you could be using NiFi Registry instead stackoverflow.com/questions/66711274/… — Sdairs
NiPyApi is on version 0.16.2-Alpha and a privately developed GitRepo, so I would rather not use that in production.... — Cribber

Cribber Cribber · Accepted Answer · 2021-03-24T12:39:16

So the problem was that the API documentation is a bit misleading... The correct API to create a new processor is process-groups/{process_group_id}/processors . It is also listed in the docs under "Process Groups" and not under "Processors" despite the description.

The following worked for me - it was necessary to adapt the json a bit though: mainly to delete any IDs of the Dev-environment.

# GET processor and parse to JSON
response = requests.get(nifi_url_dev + 'processors/' + proc_id
                            , headers=header)
processor = json.loads(response.content)


# PUT processor 
del processor["id"]  # Processor ID cannot be specified.
del processor["uri"]  # Processor ID cannot be specified.
del processor["component"]["id"]  # Processor ID cannot be specified.

del processor["component"]["parentGroupId"]  # Parent process group id should not be specified.
# If specified, the parent process group id must be the same as specified in the POST-URI.

processor['revision']['version'] = 0 # reset version
payload = json.dumps(processor).encode('utf8')
response = requests.post(nifi_url_prd + 'process-groups/' + process_group + '/processors'
                        , data=payload
                        , headers=header)

Create Nifi PutSQL Processor with Python

1 Answers