1
votes

This only happens when the system is under load. At least it is noticable then.

Postgres 8.3.14 JDK 1.6 Tomcat 6 Machine has 7GB of memory

I've noticed my postgres data directory is eating up my entire hd as well. I have postgres on a 100GB drive and it will fill it up within a few hours.

I have 10 million rows, lots of joins, mostly reads.

I'm noticing this now because my usage has increased 10x overnight.

shared_buffers = 2GB effective_cache_size = 3GB everything else if pretty much left unchanged

21:56:59,736 DEBUG BasicResourcePool:1620 - trace com.mchange.v2.resourcepool.BasicResourcePool@176b9ea [managed: 24, unused: 1, excluded: 0] (e.g. com.mchange.v2.c3p0.impl.NewPooledConnection@1c3dc66) 21:56:59,736 DEBUG BasicResourcePool:1620 - trace com.mchange.v2.resourcepool.BasicResourcePool@176b9ea [managed: 24, unused: 1, excluded: 0] (e.g. com.mchange.v2.c3p0.impl.NewPooledConnection@1c3dc66) 21:56:59,736 DEBUG BasicResourcePool:1450 - FINISHED check for expired resources. [com.mchange.v2.resourcepool.BasicResourcePool@176b9ea] 21:56:59,736 DEBUG BasicResourcePool:956 - Preparing to destroy resource: com.mchange.v2.c3p0.impl.NewPooledConnection@1e0a562 21:56:59,736 DEBUG C3P0PooledConnectionPool:468 - Preparing to destroy PooledConnection: com.mchange.v2.c3p0.impl.NewPooledConnection@1e0a562 21:56:59,736 DEBUG NewPooledConnection:564 - com.mchange.v2.c3p0.impl.NewPooledConnection@1e0a562 closed by a client. java.lang.Exception: DEBUG -- CLOSE BY CLIENT STACK TRACE at com.mchange.v2.c3p0.impl.NewPooledConnection.close(NewPooledConnection.java:564) at com.mchange.v2.c3p0.impl.NewPooledConnection.close(NewPooledConnection.java:234) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.destroyResource(C3P0PooledConnectionPool.java:470) at com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask.run(BasicResourcePool.java:958) at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547) 21:56:59,736 DEBUG C3P0PooledConnectionPool:476 - Successfully destroyed PooledConnection: com.mchange.v2.c3p0.impl.NewPooledConnection@1e0a562 21:56:59,736 DEBUG BasicResourcePool:961 - Successfully destroyed resource: com.mchange.v2.c3p0.impl.NewPooledConnection@1e0a562 21:56:59,737 DEBUG NewPooledConnection:430 - com.mchange.v2.c3p0.impl.NewPooledConnection@1e0a562 handling a throwable. org.postgresql.util.PSQLException: An I/O error occured while sending to the backend. at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:283) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:367) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:271) at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:76) at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208) at org.hibernate.loader.Loader.getResultSet(Loader.java:1953) at org.hibernate.loader.Loader.doQuery(Loader.java:802) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:274) at org.hibernate.loader.Loader.loadEntity(Loader.java:2037) at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:86) at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:76) at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:3268) at org.hibernate.event.def.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:496) at org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:477) at org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:227) at org.hibernate.event.def.DefaultLoadEventListener.lockAndLoad(DefaultLoadEventListener.java:403) at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:155) at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:1090) at org.hibernate.impl.SessionImpl.load(SessionImpl.java:1061) at org.hibernate.impl.SessionImpl.load(SessionImpl.java:1050) at com.icon.sa.common.dm.dao.BaseDao.findByIdLockedOrException(BaseDao.java:52) at com.icon.sa.service.action.auser.ItemUpdateAction.pre(ItemUpdateAction.java:28) at com.icon.sa.service.action.BaseAction.perform(BaseAction.java:59) at com.icon.sa.ws.auser.ItemResource.update(ItemResource.java:43) at sun.reflect.GeneratedMethodAccessor299.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:168) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:70) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:279) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:121) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:136) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:121) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:136) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:86) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:136) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:74) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1357) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1289) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1239) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1229) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:420) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:497) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:684) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:647) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:864) at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:579) at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1665) at java.lang.Thread.run(Thread.java:636) Caused by: java.net.SocketException: Socket closed at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:146) at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:135) at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:104) at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73) at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:259) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1620) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257) ... 58 more 21:56:59,737 DEBUG SqlUtils:85 - Attempted to convert SQLException to SQLException. Leaving it alone. [SQLState: 08006; errorCode: 0] org.postgresql.util.PSQLException: An I/O error occured while sending to the backend. at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:283) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:367) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:271) at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:76) at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208) at org.hibernate.loader.Loader.getResultSet(Loader.java:1953) at org.hibernate.loader.Loader.doQuery(Loader.java:802) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:274) at org.hibernate.loader.Loader.loadEntity(Loader.java:2037) at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:86) at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:76) at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:3268) at org.hibernate.event.def.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:496) at org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:477) at org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:227) at org.hibernate.event.def.DefaultLoadEventListener.lockAndLoad(DefaultLoadEventListener.java:403) at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:155) at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:1090) at org.hibernate.impl.SessionImpl.load(SessionImpl.java:1061) at org.hibernate.impl.SessionImpl.load(SessionImpl.java:1050) at com.icon.sa.common.dm.dao.BaseDao.findByIdLockedOrException(BaseDao.java:52) at com.icon.sa.service.action.auser.ItemUpdateAction.pre(ItemUpdateAction.java:28) at com.icon.sa.service.action.BaseAction.perform(BaseAction.java:59) at com.icon.sa.ws.auser.ItemResource.update(ItemResource.java:43) at sun.reflect.GeneratedMethodAccessor299.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:168) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:70) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:279) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:121) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:136) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:121) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:136) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:86) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:136) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:74) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1357) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1289) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1239) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1229) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:420) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:497) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:684) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:647) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:864) at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:579) at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1665) at java.lang.Thread.run(Thread.java:636) Caused by: java.net.SocketException: Socket closed at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:146) at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:135) at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:104) at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73) at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:259) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1620) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257) ... 58 more 21:56:59,738 DEBUG DefaultConnectionTester:126 - Testing a Connection in response to an Exception: org.postgresql.util.PSQLException: An I/O error occured while sending to the backend. at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:283) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:367) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:271) at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:76) at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208) at org.hibernate.loader.Loader.getResultSet(Loader.java:1953) at org.hibernate.loader.Loader.doQuery(Loader.java:802) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:274) at org.hibernate.loader.Loader.loadEntity(Loader.java:2037) at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:86) at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:76) at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:3268) at org.hibernate.event.def.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:496) at org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:477) at org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:227) at org.hibernate.event.def.DefaultLoadEventListener.lockAndLoad(DefaultLoadEventListener.java:403) at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:155) at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:1090) at org.hibernate.impl.SessionImpl.load(SessionImpl.java:1061) at org.hibernate.impl.SessionImpl.load(SessionImpl.java:1050) at com.icon.sa.common.dm.dao.BaseDao.findByIdLockedOrException(BaseDao.java:52) at com.icon.sa.service.action.auser.ItemUpdateAction.pre(ItemUpdateAction.java:28) at com.icon.sa.service.action.BaseAction.perform(BaseAction.java:59) at com.icon.sa.ws.auser.ItemResource.update(ItemResource.java:43) at sun.reflect.GeneratedMethodAccessor299.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:168) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:70) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:279) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:121) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:136) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:121) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:136) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:86) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:136) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:74) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1357) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1289) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1239) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1229) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:420) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:497) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:684) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:647) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:864) at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:579) at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1665) at java.lang.Thread.run(Thread.java:636) Caused by: java.net.SocketException: Socket closed at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:146) at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:135) at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:104) at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73) at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:259) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1620) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257) ... 58 more 21:56:59,738 DEBUG DefaultConnectionTester:194 - Connection org.postgresql.jdbc4.Jdbc4Connection@3aca failed default system-table Connection test with an Exception! org.postgresql.util.PSQLException: An I/O error occured while sending to the backend. at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:283) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:353) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:252) at org.postgresql.jdbc2.AbstractJdbc2DatabaseMetaData.getTables(AbstractJdbc2DatabaseMetaData.java:2062) at com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnectionNoQuery(DefaultConnectionTester.java:185) at com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:62) at com.mchange.v2.c3p0.impl.DefaultConnectionTester.statusOnException(DefaultConnectionTester.java:143) at com.mchange.v2.c3p0.AbstractConnectionTester.statusOnException(AbstractConnectionTester.java:82) at com.mchange.v2.c3p0.impl.NewPooledConnection.handleThrowable(NewPooledConnection.java:437) at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:93) at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208) at org.hibernate.loader.Loader.getResultSet(Loader.java:1953) at org.hibernate.loader.Loader.doQuery(Loader.java:802) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:274) at org.hibernate.loader.Loader.loadEntity(Loader.java:2037) at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:86) at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:76) at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:3268) at org.hibernate.event.def.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:496) at org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:477) at org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:227) at org.hibernate.event.def.DefaultLoadEventListener.lockAndLoad(DefaultLoadEventListener.java:403) at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:155) at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:1090) at org.hibernate.impl.SessionImpl.load(SessionImpl.java:1061) at org.hibernate.impl.SessionImpl.load(SessionImpl.java:1050) at com.icon.sa.common.dm.dao.BaseDao.findByIdLockedOrException(BaseDao.java:52) at com.icon.sa.service.action.auser.ItemUpdateAction.pre(ItemUpdateAction.java:28) at com.icon.sa.service.action.BaseAction.perform(BaseAction.java:59) at com.icon.sa.ws.auser.ItemResource.update(ItemResource.java:43) at sun.reflect.GeneratedMethodAccessor299.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:168) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:70) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:279) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:121) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:136) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:121) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:136) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:86) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:136) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:74) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1357) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1289) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1239) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1229) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:420) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:497) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:684) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:647) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:864) at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:579) at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1665) at java.lang.Thread.run(Thread.java:636) Caused by: java.io.IOException: Stream closed at sun.nio.cs.StreamEncoder.ensureOpen(StreamEncoder.java:44) at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:139) at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229) at org.postgresql.core.PGStream.flush(PGStream.java:507) at org.postgresql.core.v3.QueryExecutorImpl.sendSync(QueryExecutorImpl.java:1107) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:256) ... 64 more

Here is my config

<property name="c3p0.acquire_increment">1</property> 
    <property name="c3p0.idle_test_period">100</property> <!-- seconds --> 
    <property name="c3p0.max_size">250</property> 
    <property name="c3p0.max_statements">0</property> 
    <property name="c3p0.min_size">10</property> 
    <property name="c3p0.timeout">100</property> <!-- seconds --> 

<!-- Configuring Connection Testing -->
    <!-- Create a test table to query for on checkout and checkin. A test table
    is faster than to use the getTables default cmd. -->
    <property name="automaticTestTable">c3p0_test</property>
    <property name="checkoutTimeout">0</property>
    <!-- This tests to make sure the connection is still valid (needed for high load system). -->
    <property name="testConnectionOnCheckout">true</property>
    <property name="testConnectionOnCheckin">true</property>
    <!-- Configuring Recovery From Database Outages -->
    <property name="acquireRetryAttempts">0</property>
    <property name="acquireRetryDelay">1000</property>
    <property name="breakAfterAcquireFailure">false</property>
    <!-- This is a safeguard to cleanup any open, abandoned connections.
    These are connections that are opened but never closed. Debug on server if this ever needs
    to cleanup.
    We can test without this feature (its discouraged to use)     -->
    <property name="unreturnedConnectionTimeout">300</property>
    <property name="debugUnreturnedConnectionStackTraces">true</property>
1
I bolded the actual errors, maybe that will give you a clue. The likelihood that anyone here knows what the problem is without knowing more details about your environment seems pretty close to zero.Robert Harvey
Visit the PostgreSQL mailing lists and you are more likely to get help on this. postgresql.org/community/listsKuberchaun
i updated the question to include more environment detailskgibbon
Any errors in the PostgreSQL log? Also regarding the bloat, what is your postgresql version?Chris Travers

1 Answers

0
votes

These sort of things are usually caused by back-end crashes. It is likely there is a bad extension or library that is making a mess of things. What you'd want to do is check the PostgreSQL logs for errors, or capture queries and try to see where this happens. If that doesn't trace the problem it could be failing hardware.

On a database server this sort of thing is relatively urgent in the sense that it may interfere with data being saved or even result in database corruption. My recommendation is to seek professional help as soon as possible if you seen an issue like this.