We are using mysql(5.1.73), jBoss EAP 6.2 and Java. Application works fine in Development without error, but in production, we get below error. Say around 5-10% of connections are failing. Can you please help identify issue?
We followed another post, but that didn't help https://developer.jboss.org/thread/231941?start=0&tstart=0&_sscc=t
<validation>
<check-valid-connection-sql>SELECT 1</check-valid-connection-sql>
<background-validation>true</background-validation>
<background-validation-millis>5000</background-validation-millis>
<exception-sorter class-name="org.jboss.jca.adapters.jdbc.extensions.mysql.MySQLExceptionSorter"/>
</validation>
Can't add comment - Editing question, @Dukefirehawk yes, did check jBoss EAP console, in Development environment and the connections are getting closed. Only the min pool size remains..
Below is the complete stack trace of error.
SEVERE Communications link failure
The last packet successfully received from the server was 383,746 milliseconds ago. The last packet sent successfully to the server was 3,022 milliseconds ago.: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 383,746 milliseconds ago. The last packet sent successfully to the server was 3,022 milliseconds ago.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [rt.jar:1.7.0_75]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) [rt.jar:1.7.0_75]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [rt.jar:1.7.0_75]
at java.lang.reflect.Constructor.newInstance(Constructor.java:526) [rt.jar:1.7.0_75]
at com.mysql.jdbc.Util.handleNewInstance(Util.java:408) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1137) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3715) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3604) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4149) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2615) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2776) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2834) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.ConnectionImpl.setTransactionIsolation(ConnectionImpl.java:5674) [mysql-connector-java-5.1.36.jar:5.1.36]
at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.setJdbcTransactionIsolation(BaseWrapperManagedConnection.java:1045)
at org.jboss.jca.adapters.jdbc.WrappedConnection.setTransactionIsolation(WrappedConnection.java:906)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at org.jboss.modcluster.container.jbossweb.JBossWebContext$RequestListenerValve.event(JBossWebContext.java:91)
at org.jboss.modcluster.container.jbossweb.JBossWebContext$RequestListenerValve.invoke(JBossWebContext.java:72)
at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:488) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_75]
Caused by: java.net.SocketException: Connection timed out
at java.net.SocketInputStream.socketRead0(Native Method) [rt.jar:1.7.0_75]
at java.net.SocketInputStream.read(SocketInputStream.java:152) [rt.jar:1.7.0_75]
at java.net.SocketInputStream.read(SocketInputStream.java:122) [rt.jar:1.7.0_75]
at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:112) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:159) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:187) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3158) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3615) [mysql-connector-java-5.1.36.jar:5.1.36]
... 29 more
CONNECTION IN ADD USERorg.jboss.jca.adapters.jdbc.jdk6.WrappedConnectionJDK6@780618f4
WARN [org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory] Exception trying to create stale connection checker (disabling) 5000: java.lang.Exception: Unable to load: 5000
at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnectionFactory.loadPlugin(BaseWrapperManagedConnectionFactory.java:1025)
at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnectionFactory.isStaleConnection(BaseWrapperManagedConnectionFactory.java:1148)
at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.connectionError(BaseWrapperManagedConnection.java:588)
at org.jboss.jca.adapters.jdbc.WrappedConnection.checkException(WrappedConnection.java:1640)
at org.jboss.jca.adapters.jdbc.WrappedConnection.createStatement(WrappedConnection.java:309)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at org.jboss.modcluster.container.jbossweb.JBossWebContext$RequestListenerValve.event(JBossWebContext.java:91)
at org.jboss.modcluster.container.jbossweb.JBossWebContext$RequestListenerValve.invoke(JBossWebContext.java:72)
at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:488) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_75]
WARN [org.jboss.jca.core.connectionmanager.listener.NoTxConnectionListener] IJ000305: Connection error occured: org.jboss.jca.core.connectionmanager.listener.NoTxConnectionListener@13cf00aa[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@1d171e0a connection handles=1 lastUse=1452721124659 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@58b53171 pool internal context=SemaphoreArrayListManagedConnectionPool@5fc11f36[pool=#######]]: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [rt.jar:1.7.0_75]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) [rt.jar:1.7.0_75]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [rt.jar:1.7.0_75]
at java.lang.reflect.Constructor.newInstance(Constructor.java:526) [rt.jar:1.7.0_75]
at com.mysql.jdbc.Util.handleNewInstance(Util.java:408) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.Util.getInstance(Util.java:383) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1023) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:997) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:983) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:928) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.ConnectionImpl.throwConnectionClosedException(ConnectionImpl.java:1323) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.ConnectionImpl.checkClosed(ConnectionImpl.java:1315) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.ConnectionImpl.createStatement(ConnectionImpl.java:2698) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.ConnectionImpl.createStatement(ConnectionImpl.java:2680) [mysql-connector-java-5.1.36.jar:5.1.36]
at org.jboss.jca.adapters.jdbc.WrappedConnection.createStatement(WrappedConnection.java:305)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at org.jboss.modcluster.container.jbossweb.JBossWebContext$RequestListenerValve.event(JBossWebContext.java:91)
at org.jboss.modcluster.container.jbossweb.JBossWebContext$RequestListenerValve.invoke(JBossWebContext.java:72)
at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:488) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_75]
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 383,746 milliseconds ago. The last packet sent successfully to the server was 3,022 milliseconds ago.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [rt.jar:1.7.0_75]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) [rt.jar:1.7.0_75]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [rt.jar:1.7.0_75]
at java.lang.reflect.Constructor.newInstance(Constructor.java:526) [rt.jar:1.7.0_75]
at com.mysql.jdbc.Util.handleNewInstance(Util.java:408) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1137) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3715) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3604) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4149) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2615) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2776) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2834) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.ConnectionImpl.setTransactionIsolation(ConnectionImpl.java:5674) [mysql-connector-java-5.1.36.jar:5.1.36]
at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.setJdbcTransactionIsolation(BaseWrapperManagedConnection.java:1045)
at org.jboss.jca.adapters.jdbc.WrappedConnection.setTransactionIsolation(WrappedConnection.java:906)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
... 15 more
Caused by: java.net.SocketException: Connection timed out
at java.net.SocketInputStream.socketRead0(Native Method) [rt.jar:1.7.0_75]
at java.net.SocketInputStream.read(SocketInputStream.java:152) [rt.jar:1.7.0_75]
at java.net.SocketInputStream.read(SocketInputStream.java:122) [rt.jar:1.7.0_75]
at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:112) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:159) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:187) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3158) [mysql-connector-java-5.1.36.jar:5.1.36]
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3615) [mysql-connector-java-5.1.36.jar:5.1.36]
... 29 more
