How to update flows from dev to prod with state

Question

I have a nifi flow it has keeps some state with the ListS3 processor. I have a dev instance and a prod instance. I want some options of deploying from dev to prod where the state is kept and where I don't manually have to go in and change all the processor's and process groups.

It seems like this can't be done with templates? Based on the following stackoverflow question:

how does NIFI listfile maintains its timestamp?

edit: Just so there is no misunderstanding I want to keep prod state when deploying.

Answer 1

It sounds like you aren't using NiFi registry, so you're downloading a flow template and then importing it. This can't preserve state, as it's not the same flow.

You should be using NiFi Registry to version control your flows, which supports this Dev -> Prod workflow.

Build your flow in Dev NiFi, version to Registry.

In prod, add a new Process Group and select the Import option when it asks you for a name. You'll be able to pick your versioned flow.

Run your flow so that it stores some state. View the processors state to verify.

Now update the flow in Dev, and commit the local change to Registry.

Then, update the flow in Prod to the latest version from Registry. It will preserve state on the stateful processor.

For detailed steps on installing & using Registry, see these links:

https://nifi.apache.org/docs/nifi-registry-docs/html/getting-started.html

https://pierrevillard.com/2018/04/09/automate-workflow-deployment-in-apache-nifi-with-the-nifi-registry/

https://alasdairb.com/2021/03/22/nifi-in-production-nifi-registry/

https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.2.0/versioning-a-dataflow/content/connecting-to-a-nifi-registry.html

https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.4.0/getting-started-with-nifi-registry/content/import-a-versioned-flow.html

https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.4.0/getting-started-with-nifi-registry/content/save-changes-to-a-versioned-flow.html

https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.4.0/getting-started-with-nifi-registry/content/start-version-control-on-a-process-group.html