Run NiFi on a linux machine

Question

I just downloaded nifi-0.5.1 for linux.

When trying to run Nifi using the command: ./nifi.sh run

I get the following exception in nifi-app.log:

2016-03-13 05:29:42,959 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down. org.apache.nifi.web.NiFiCoreException: Unable to start Flow Controller. at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:99) ~[na:na] at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:800) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529] at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:444) ~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529] at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:791) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529] at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:294) ~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529] at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349) ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529] at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342) ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529] at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529] at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505) ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529] at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) ~[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529] at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) ~[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529] at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529] at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) ~[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529] at org.eclipse.jetty.server.Server.start(Server.java:387) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529] at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) ~[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529] at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529] at org.eclipse.jetty.server.Server.doStart(Server.java:354) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529] at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:663) ~[nifi-jetty-0.5.1.jar:0.5.1] at org.apache.nifi.NiFi.(NiFi.java:137) [nifi-runtime-0.5.1.jar:0.5.1] at org.apache.nifi.NiFi.main(NiFi.java:227) [nifi-runtime-0.5.1.jar:0.5.1] Caused by: java.nio.file.FileSystemException: ./flowfile_repository/partition-61/2.journal: Too many open files at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) ~[na:1.7.0_45] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[na:1.7.0_45] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[na:1.7.0_45] at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[na:1.7.0_45] at java.nio.file.Files.newByteChannel(Files.java:315) ~[na:1.7.0_45] at java.nio.file.Files.newByteChannel(Files.java:361) ~[na:1.7.0_45] at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:380) ~[na:1.7.0_45] at java.nio.file.Files.newInputStream(Files.java:106) ~[na:1.7.0_45] at org.wali.MinimalLockingWriteAheadLog$Partition.createDataInputStream(MinimalLockingWriteAheadLog.java:932) ~[nifi-write-ahead-log-0.5.1.jar:0.5.1] at org.wali.MinimalLockingWriteAheadLog$Partition.getRecoveryStream(MinimalLockingWriteAheadLog.java:947) ~[nifi-write-ahead-log-0.5.1.jar:0.5.1] at org.wali.MinimalLockingWriteAheadLog$Partition.getNextRecoverableTransactionId(MinimalLockingWriteAheadLog.java:973) ~[nifi-write-ahead-log-0.5.1.jar:0.5.1] at org.wali.MinimalLockingWriteAheadLog.recoverFromEdits(MinimalLockingWriteAheadLog.java:419) ~[nifi-write-ahead-log-0.5.1.jar:0.5.1] at org.wali.MinimalLockingWriteAheadLog.recoverRecords(MinimalLockingWriteAheadLog.java:293) ~[nifi-write-ahead-log-0.5.1.jar:0.5.1] at org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.loadFlowFiles(WriteAheadFlowFileRepository.java:323) ~[nifi-framework-core-0.5.1.jar:0.5.1] at org.apache.nifi.controller.FlowController.initializeFlow(FlowController.java:609) ~[nifi-framework-core-0.5.1.jar:0.5.1] at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:641) ~[nifi-framework-core-0.5.1.jar:0.5.1] at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:479) ~[nifi-framework-core-0.5.1.jar:0.5.1] at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:79) ~[na:na] ... 22 common frames omitted 2016-03-13 05:29:42,961 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...

I would appreciate your help,

Answer 1

By default NiFi accesses quite a lot of files at startup due to its classloader isolation approach and its function as a dataflow app. Please see the administration guide where it describes how to set the OS limit on number of open file handles higher. The default OS file handle limit in linux is usually 1024 and this is quite a ways too low for what NiFi does.

https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#configuration-best-practices