JVM Crash with SIGSEGV - problematic frame - libzip.so

Question

JVM crashes strangely from time to time. The problematic frame in JVM crash log is libzip.so and error is SIGSEGV. This occurs in Tomcat 7 and JDK 7

Here is the Stack Trace for reference:

Stack: [0x00007f68b999c000,0x00007f68b9a9d000], sp=0x00007f68b9a99770,

free space=1013k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) C [libzip.so+0x8099] deflate_slow+0x49 C

[libzip.so+0x7433] deflate+0x163 C [libzip.so+0x3049]

Java_java_util_zip_Deflater_deflateBytes+0x269 j java.util.zip.Deflater.deflateBytes(J[BIII)I+0 J 3307 C2 java.util.zip.GZIPOutputStream.finish()V (135 bytes) @

org.apache.coyote.http11.filters.FlushableGZIPOutputStream.finish()V+9 j org.apache.coyote.http11.filters.GzipOutputFilter.end()J+26 j org.apache.coyote.http11.AbstractOutputBuffer.endRequest()V+43 J 2372 C2

org.apache.coyote.http11.AbstractHttp11Processor.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V (602 bytes) @ 0x00007f6a4d759c84 [0x00007f6a4d759a80+0x204] J 2733 C2 org.apache.coyote.Response.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V

(37 bytes) @ 0x00007f6a4d8b9ad0 [0x00007f6a4d8b9a80+0x50] j org.apache.coyote.Response.finish()V+5 j org.apache.catalina.connector.OutputBuffer.close()V+138 j org.apache.catalina.connector.CoyoteWriter.close()V+4 j org.apache.catalina.core.ApplicationDispatcher.doForward(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+467 j org.apache.catalina.core.ApplicationDispatcher.forward(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+55 j org.apache.struts.action.RequestProcessor.doForward(Ljava/lang/String;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+56 j org.apache.struts.tiles.TilesRequestProcessor.doForward(Ljava/lang/String;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+23 j org.apache.struts.tiles.TilesRequestProcessor.processTilesDefinition(Ljava/lang/String;ZLjavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)Z+385 j org.apache.struts.tiles.TilesRequestProcessor.processForwardConfig(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Lorg/apache/struts/config/ForwardConfig;)V+74 j org.apache.struts.action.RequestProcessor.process(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+267 j org.apache.struts.action.ActionServlet.process(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+40 j org.apache.struts.action.ActionServlet.doGet(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+3 j javax.servlet.http.HttpServlet.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+35 j javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+30 J 2406 C2 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (624 bytes) @ 0x00007f6a4d7760e4 [0x00007f6a4d775da0+0x344] J 2407 C2 org.apache.catalina.core.ApplicationFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (105 bytes) @ 0x00007f6a4d5b2154 [0x00007f6a4d5b2120+0x34] j org.apache.tomcat.websocket.server.WsFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V+21 J 2406 C2 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (624 bytes) @ 0x00007f6a4d775f28 [0x00007f6a4d775da0+0x188] J 2407 C2 org.apache.catalina.core.ApplicationFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (105 bytes) @ 0x00007f6a4d5b2154 [0x00007f6a4d5b2120+0x34] j org.apache.catalina.core.ApplicationDispatcher.invoke(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Lorg/apache/catalina/core/ApplicationDispatcher$State;)V+393 j

org.apache.catalina.core.ApplicationDispatcher.processRequest(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Lorg/apache/catalina/core/ApplicationDispatcher$State;)V+98

This issue occurs only after migrating to JDK 7 + Tomcat7. It was working fine in the lower versions. Any help is much appreciated.

Answer 1

Since there is a jni call in stack trace, try to run jvm with -Xcheck:jni option ( I am not sure if it will detect some errors in your case).

This crash occurs while compressing output stream in tomcat filter. You can disable compression in connector ( here )

Also as I see, error occurs in deflate_slow method, you can turn off tomcat GzipOutputFilter and create your custom filter with DeflaterOutputStream and set compression level between 1-3 ( In this case deflate_fast method will be choosen in hotspot)

Answer 2

Please use -Dsun.zip.disableMemoryMapping=true to avoid such crash, JDK9 has better mechanism to handle such crashes. If it is feasible please upgrade to use JDK9