Database connection is closed in Jboss AS

2
votes

I have an application that runs behind JBoss Application Server. The client handles database operations via server. EJB beans handle database operations using entity manager.

After a time, I see that database connections are closed automatically. Then the client is not able to request anything from server. Exception is thrown in each request. I am sending you starting point for exception. Maybe there is a time out for database connection. But Client is not an idle process. It is a simulator and always requests data from server and sends new data to persistent to server.

Here exception 

2014-01-14 20:37:35,764 INFO  [org.jboss.ejb3.async.impl.interceptor.AsynchronousClientInterceptor] (Thread-356 (group:HornetQ-client-global-threads-1610282042)) OID: null
2014-01-14 20:37:39,161 INFO  [org.jboss.ejb3.async.impl.interceptor.AsynchronousClientInterceptor] (Thread-356 (group:HornetQ-client-global-threads-1610282042)) OID: null
2014-01-14 20:38:12,572 INFO  [org.jboss.ejb3.async.impl.interceptor.AsynchronousClientInterceptor] (Thread-337 (group:HornetQ-client-global-threads-1610282042)) OID: null
2014-01-14 20:39:00,949 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA-12117 TransactionReaper::check timeout for TX 0:ffffc0a80127:126a:52d5345b:16fe4b in state  RUN
2014-01-14 20:39:00,950 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA-12095 Abort of action id 0:ffffc0a80127:126a:52d5345b:16fe4b invoked while multiple threads active within it.
2014-01-14 20:39:00,950 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA-12108 CheckedAction::check - atomic action 0:ffffc0a80127:126a:52d5345b:16fe4b aborting with 1 threads active!
2014-01-14 20:39:03,977 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA-12117 TransactionReaper::check timeout for TX 0:ffffc0a80127:126a:52d5345b:16fe4b in state  CANCEL
2014-01-14 20:39:03,979 ERROR [org.hibernate.event.def.AbstractFlushingEventListener] (pool-4-thread-9) Could not synchronize database state with session: org.hibernate.SessionException: Session is closed!
        at org.hibernate.impl.AbstractSessionImpl.errorIfClosed(AbstractSessionImpl.java:72) [:3.6.0.Final]
        at org.hibernate.impl.SessionImpl.getBatcher(SessionImpl.java:305) [:3.6.0.Final]
        at org.hibernate.persister.collection.AbstractCollectionPersister.recreate(AbstractCollectionPersister.java:1217) [:3.6.0.Final]
        at org.hibernate.action.CollectionRecreateAction.execute(CollectionRecreateAction.java:58) [:3.6.0.Final]
        at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:273) [:3.6.0.Final]
        at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:265) [:3.6.0.Final]
        at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:188) [:3.6.0.Final]
        at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321) [:3.6.0.Final]
        at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:51) [:3.6.0.Final]
        at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1216) [:3.6.0.Final]
        at org.hibernate.ejb.AbstractEntityManagerImpl.flush(AbstractEntityManagerImpl.java:795) [:3.6.0.Final]
        at org.jboss.jpa.impl.tx.TransactionScopedEntityManager.flush(TransactionScopedEntityManager.java:235) [:2.0.0]
        at com.siemens.siguard.simulation.ejb.SimulationResultGraphPersitenceBean.persistGraph(SimulationResultGraphPersitenceBean.java:32) [:]
        at sun.reflect.GeneratedMethodAccessor515.invoke(Unknown Source) [:1.6.0_25]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [:1.6.0_25]
        at java.lang.reflect.Method.invoke(Method.java:597) [:1.6.0_25]
        at org.jboss.aop.joinpoint.MethodInvocation.invokeTarget(MethodInvocation.java:122) [jboss-aop.jar:2.2.1.GA]
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111) [jboss-aop.jar:2.2.1.GA]
        at org.jboss.ejb3.interceptors.container.ContainerMethodInvocationWrapper.invokeNext(ContainerMethodInvocationWrapper.java:72) [:1.1.3]
        at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.invoke(InterceptorSequencer.java:76) [:1.1.3]
        at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.aroundInvoke(InterceptorSequencer.java:62) [:1.1.3]
        at sun.reflect.GeneratedMethodAccessor377.invoke(Unknown Source) [:1.6.0_25]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [:1.6.0_25]
        at java.lang.reflect.Method.invoke(Method.java:597) [:1.6.0_25]
        at org.jboss.aop.advice.PerJoinpointAdvice.invoke(PerJoinpointAdvice.java:174) [jboss-aop.jar:2.2.1.GA]
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.1.GA]
        at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.fillMethod(InvocationContextInterceptor.java:74) [:1.1.3]
        at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_fillMethod_500571876.invoke(InvocationContextInterceptor_z_fillMethod_500571876.java) [:]
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.1.GA]
        at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.setup(InvocationContextInterceptor.java:90) [:1.1.3]
        at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_setup_500571876.invoke(InvocationContextInterceptor_z_setup_500571876.java) [:]
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.1.GA]
        at org.jboss.ejb3.async.impl.interceptor.AsynchronousServerInterceptor.invoke(AsynchronousServerInterceptor.java:128) [:1.7.17]
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.1.GA]
        at org.jboss.ejb3.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:62) [:1.7.17]
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.1.GA]
        at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:56) [:1.7.17]
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.1.GA]
        at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47) [:1.7.17]
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.1.GA]
        at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42) [:1.0.3]
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.1.GA]
        at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:68) [:1.7.17]
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.1.GA]
        at org.jboss.ejb3.core.context.SessionInvocationContextAdapter.proceed(SessionInvocationContextAdapter.java:95) [:1.7.17]
        at org.jboss.ejb3.tx2.impl.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:247) [:0.0.1]
        at org.jboss.ejb3.tx2.impl.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:392) [:0.0.1]
        at org.jboss.ejb3.tx2.impl.CMTTxInterceptor.invoke(CMTTxInterceptor.java:211) [:0.0.1]
        at org.jboss.ejb3.tx2.aop.CMTTxInterceptorWrapper.invoke(CMTTxInterceptorWrapper.java:52) [:0.0.1]
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.1.GA]
        at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76) [:1.0.0.GA]
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.1.GA]
        at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42) [:1.0.3]
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.1.GA]
        at org.jboss.ejb3.security.Ejb3AuthenticationInterceptorv2.invoke(Ejb3AuthenticationInterceptorv2.java:182) [:1.7.17]
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.1.GA]
        at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:41) [:1.7.17]
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.1.GA]
        at org.jboss.ejb3.BlockContainerShutdownInterceptor.invoke(BlockContainerShutdownInterceptor.java:67) [:1.7.17]
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.1.GA]
        at org.jboss.ejb3.core.context.CurrentInvocationContextInterceptor.invoke(CurrentInvocationContextInterceptor.java:47) [:1.7.17]
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.1.GA]
        at org.jboss.aspects.currentinvocation.CurrentInvocationInterceptor.invoke(CurrentInvocationInterceptor.java:67) [:1.0.1]
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.1.GA]
        at org.jboss.ejb3.interceptor.EJB3TCCLInterceptor.invoke(EJB3TCCLInterceptor.java:86) [:1.7.17]
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.1.GA]
        at org.jboss.ejb3.session.SessionSpecContainer.invoke(SessionSpecContainer.java:323) [:1.7.17]
        at org.jboss.ejb3.session.SessionSpecContainer.invoke(SessionSpecContainer.java:156) [:1.7.17]
        at org.jboss.ejb3.session.SessionSpecContainer$AsyncInvocationTask.proceed(SessionSpecContainer.java:226) [:1.7.17]
        at org.jboss.ejb3.async.spi.AsyncInvocationTaskBase.call(AsyncInvocationTaskBase.java:90) [:1.0.0-alpha-9]
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [:1.6.0_25]
        at java.util.concurrent.FutureTask.run(FutureTask.java:138) [:1.6.0_25]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) [:1.6.0_25]
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [:1.6.0_25]
        at java.util.concurrent.FutureTask.run(FutureTask.java:138) [:1.6.0_25]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98) [:1.6.0_25]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206) [:1.6.0_25]
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_25]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_25]
        at java.lang.Thread.run(Thread.java:662) [:1.6.0_25]

this is java function that is written in log.@Asynchronous annotation can lead to this problem?

@TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)
@Asynchronous
public void persistGraph(ResultGraph graph)  {
    try {
        ResultGraphService service = new ResultGraphService(em);
        service.create(graph);
        em.flush();
    } catch (Exception ex) {
        throw new Exception("Error persisting graph", ex);
    }
}
1
javahibernatejakarta-eejboss
Have you tried to configure the datasource pool to do test on borrow? (or whatever is called in JBoss land) – Augusto
How can I do this? I think you mention about ds.xml Which paramaters I will use? – user725455
Are you talking about these parameters? <blocking-timeout-millis>30000</blocking-timeout-millis> <idle-timeout-minutes>5</idle-timeout-minutes> – user725455
No, it's a query that JBoss will run before returning a connection, for example in oracle this query is usually select 1 from dual. See this other question: stackoverflow.com/questions/128527/… – Augusto

1 Answers

2
votes

You may have two problems going on, the first of which is that your transaction is not committing within the timeout period allowed by the JBoss transaction coordinator (ARJUNA).

However, if your transaction is just long-running, then this suggestion may help. Look in your JBoss setup file (on Linux it would be somewhere like /etc/jbossas/standalone/standalone.xml in a non-clustered environment) and look for this section:

    <subsystem xmlns="urn:jboss:domain:transactions:1.3">
        <core-environment>
            <process-id>
                <uuid/>
            </process-id>
        </core-environment>
        <recovery-environment socket-binding="txn-recovery-environment" status-socket-binding="txn-status-manager"/>
        <coordinator-environment default-timeout="300"/>
    </subsystem>

Notice the default-timeout attribute of the <coordinator-environment>. By default it is 300 seconds. Now, 5 minutes should be plenty of time for a transaction to run, but in long-running, heavy batch processing, this limit could be exceeded. If you suspect this is your problem, increase this parameter (which is in units of seconds), and bounce JBoss (you can also do this through the JBoss management console, but I've found in development it's just easier to modify the standalone.xml and bounce JBoss).

HTH