Tomcat too many files open (tomcat-users.xml)

4
votes

Earlier today I visited a site that I recently launched and the page was just filled with a big ugly jasperexception: java.io.filenotfoundexception: /opt/tomcat/webapps/root/web-inf/lib/jstl-1.2.jar (too many open files). I checked my server logs and tomcat logs and they were full of the same error. I increased tomcat's max open file limit as a temporary solution to keep the site running, but it's quickly filling up again. I've launched other sites with the same methods I used for this one, and never had this problem before. The only thing I did different this time was use Mobirise to make the base layout for the front-end, so I'm not ruling out that some scripts used by Mobirise could be the culprit. But when I get the list of files that tomcat has open, I see that it's got hundreds of copies of tomcat-users.xml open, and it just keeps opening more and more.

...
java    319 tomcat  935r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  936r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  937r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  938r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  939r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  940r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  941r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  942r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  943r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  944r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  945r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  946r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  947r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  948r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  949r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  950r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  951r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  952r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  953r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  954r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  955r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  956r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  957r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  958r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  959r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  960r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  961r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  962r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  963r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
java    319 tomcat  964r      REG              253,1     2164 263714 /opt/tomcat/conf/tomcat-users.xml
...

I don't understand why it's doing this. I've been debugging, scanning through my code, and scouring the internet for issues where tomcat keeps opening too many files, but they all just say "increase the max file limit." None of them address the root of the problem or mention anything about tomcat just infinitely opening copies of a single file.

[EDIT]: Here is the full stack trace of the FileNotFoundException:

28-Nov-2018 03:51:33.857 SEVERE [http-nio-8080-exec-5] org.apache.catalina.core.ApplicationDispatcher.invoke Servlet.service() for servlet [jsp] threw exception
 java.io.FileNotFoundException: /opt/tomcat/webapps/ROOT/WEB-INF/lib/jstl-1.2.jar (Too many open files)
        at java.util.zip.ZipFile.open(Native Method)
        at java.util.zip.ZipFile.<init>(ZipFile.java:225)
        at java.util.zip.ZipFile.<init>(ZipFile.java:155)
        at java.util.jar.JarFile.<init>(JarFile.java:166)
        at java.util.jar.JarFile.<init>(JarFile.java:130)
        at org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:164)
        at org.apache.tomcat.util.scan.JarFileUrlJar.<init>(JarFileUrlJar.java:65)
        at org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49)
        at org.apache.tomcat.util.descriptor.tld.TldResourcePath.openJar(TldResourcePath.java:134)
        at org.apache.jasper.compiler.TagLibraryInfoImpl.<init>(TagLibraryInfoImpl.java:125)
        at org.apache.jasper.compiler.Parser.parseTaglibDirective(Parser.java:431)
        at org.apache.jasper.compiler.Parser.parseDirective(Parser.java:489)
        at org.apache.jasper.compiler.Parser.parseElements(Parser.java:1445)
        at org.apache.jasper.compiler.Parser.parse(Parser.java:144)
        at org.apache.jasper.compiler.ParserController.doParse(ParserController.java:244)
        at org.apache.jasper.compiler.ParserController.parse(ParserController.java:105)
        at org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:206)
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:385)
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:362)
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:346)
        at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:603)
        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:400)
        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:386)
        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:330)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:712)
        at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:459)
        at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:384)
        at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:312)
        at kom.kitten.gingerpage.servlets.HomeServlet.doGet(HomeServlet.java:16)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:634)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at com.googlecode.objectify.ObjectifyFilter.doFilter(ObjectifyFilter.java:48)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:668)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:791)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1417)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)

28-Nov-2018 03:51:33.859 SEVERE [http-nio-8080-exec-5] org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service() for servlet [kom.kitten.gingerpage.servlets.HomeServlet] in conte$
 java.io.FileNotFoundException: /opt/tomcat/webapps/ROOT/WEB-INF/lib/jstl-1.2.jar (Too many open files)
        at java.util.zip.ZipFile.open(Native Method)
        at java.util.zip.ZipFile.<init>(ZipFile.java:225)
        at java.util.zip.ZipFile.<init>(ZipFile.java:155)
        at java.util.jar.JarFile.<init>(JarFile.java:166)
        at java.util.jar.JarFile.<init>(JarFile.java:130)
        at org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:164)
        at org.apache.tomcat.util.scan.JarFileUrlJar.<init>(JarFileUrlJar.java:65)
        at org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49)
        at org.apache.tomcat.util.descriptor.tld.TldResourcePath.openJar(TldResourcePath.java:134)
        at org.apache.jasper.compiler.TagLibraryInfoImpl.<init>(TagLibraryInfoImpl.java:125)
        at org.apache.jasper.compiler.Parser.parseTaglibDirective(Parser.java:431)
        at org.apache.jasper.compiler.Parser.parseDirective(Parser.java:489)
        at org.apache.jasper.compiler.Parser.parseElements(Parser.java:1445)
        at org.apache.jasper.compiler.Parser.parse(Parser.java:144)
        at org.apache.jasper.compiler.ParserController.doParse(ParserController.java:244)
        at org.apache.jasper.compiler.ParserController.parse(ParserController.java:105)
        at org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:206)
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:385)
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:362)
        at org.apache.jasper.compiler.Compiler.compile(Compiler.java:346)
        at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:603)
        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:400)
        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:386)
        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:330)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:712)
        at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:459)
        at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:384)
        at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:312)
        at kom.kitten.gingerpage.servlets.HomeServlet.doGet(HomeServlet.java:16)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:634)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at com.googlecode.objectify.ObjectifyFilter.doFilter(ObjectifyFilter.java:48)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:668)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:791)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1417)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)

This is the same stack trace that it logs every time the error occurs (i.e. every time someone attempts to load a page on the site while there are too many files open in tomcat)

3
javatomcat
You're correct - the problem isn't "too many open files". The REAL question is "who's opening them - and why?" Q: Any ideas? SUGGESTION: Try to get a stack trace, and copy it into your question. That might help isolate the culprit. - paulsm4
Please show the relevant code. - jww
@jww I don't know which code in my application (if any) is relevant because I don't have any idea what's causing it. I don't think my code is the problem. I wrote very little code for this site, it's a pretty basic site, and the code I did write is code that I've used numerous times before and never had this problem with. If any code were causing this, it would be something in the code used by Mobirise, but that encompasses entire libraries, way too much to sift through by hand. - Kat
I think the question is OK here but you need to do some debugging and show the relevant code. If you can't then maybe you should move the question to Super User or Unix & Linux Stack Exchange. Server Fault would probably be the best fit but they have the "professional admin" requirement. - jww
Hi, Kat! I am sure you are not aware of our our format here, but SO is a Q&A site. This means that your question should remain separate from your answer. Feel free to add your answer below if you like. You can also get help with how our site works by looking at the help center. Thanks! - K.Dᴀᴠɪs

3 Answers

5
votes

This was caused by a bug in Tomcat 9.0.13 which has been fixed in Tomcat 9.0.14.

3
votes

I just ran into the same issue after moving an application from a tomcat7 instance to a tomcat 9 instance. I have a feeling this may be a tomcat bug. I removed the tag 

<Resource pathname="conf/tomcat-users.xml" 
factory="org.apache.catalina.users.MemoryUserDatabaseFactory" 
type="org.apache.catalina.UserDatabase" 
auth="Container" name="UserDatabase"/>

in the GlobalNamingResources tag in $CATALINA_HOME/conf/server.xml and after a restart the issue resolved itself. So I'm guessing the issue lies with Tomcat's management app.

0
votes

If you have applications deploy on Tomcat. First, You must make sure that when you open a file you close it explicitly. You can use the following command in order to see the open files in linux:

lsof -p <process Id of Tomcat>