how does NIFI listfile maintains its timestamp?

Question

i am working on NiFi in which i have used processors such as list, fetch and others. The list file processer lists the file as and when new files arrives in the mentioned directory system. till now, i have done processing of one year of data files that is from the year january 2016 till date.

now i want to export flow.xml file to the new environment and continue the processing. when i start the list processor, it starts from the beginning i.e 2016 january. is there anyway to continue the listing processor where it left in the previous environment processing.

Bryan Bende Bryan Bende · Accepted Answer · 2017-05-04T13:33:37

In a standalone NiFi instance, the state is being stored by the local state provider configured in state-management.xml:

<local-provider>
        <id>local-provider</id>
        <class>org.apache.nifi.controller.state.providers.local.WriteAheadLocalStateProvider</class>
        <property name="Directory">./state/local</property>
        <property name="Always Sync">false</property>
        <property name="Partitions">16</property>
        <property name="Checkpoint Interval">2 mins</property>
    </local-provider>

You can see that the location on disk is in ./state/local so you would need to copy this directory structure and everything under it to the same location on the other NiFi instance.

Also, this only works if you are copying the flow.xml.gz from one instance to the other because state is stored based on the UUID of the component, so you need the UUIDs to stay the same. If you use templates to move the flow then you get new UUIDs and the state won't line up.

how does NIFI listfile maintains its timestamp?

1 Answers