my web site gets down on tomcat's out of memory exception

1
votes

I have apache 2 + mod_jk + tomcat setup on Redhat. Just recently started having java.lang.OutOfMemoryError: unable to create new native thread exceptions which cause tomcat hang.

catalina.sh:

JAVA_OPTS="-Xms1256m -Xmx2000m"
export JAVA_OPTS
CATALINA_OPTS="-Xms1024m -Xmx2000m"
export CATALINA_OPTS
JAVA_HOME=/data/app/jdk/jdk1.5.0_11
JRE_HOME=/data/app/jdk/jdk1.5.0_11
export JAVA_HOME
export JRE_HOME

Is the memory settings correct ? I have 4GM available RAM.

Tomcat error log:

2010-11-14 00:40:13,726 ERROR org.apache.catalina.core.StandardWrapperValve [invoke] - Servlet.service() for servlet action threw exception
java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:574)

Mod_jk log entries at time when tomcat gets down:

[Sun Nov 14 00:59:52 2010] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (127.0.0.1:8009), err=-104
[Sun Nov 14 00:59:52 2010] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Sun Nov 14 01:00:01 2010] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (127.0.0.1:8009), err=-104
[Sun Nov 14 01:00:01 2010] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Sun Nov 14 01:00:01 2010] [error] ajp_service::jk_ajp_common.c (1758): Error connecting to tomcat. Tomcat is probably not started or is listening on the wrong port. worker=worker1 failed
[Sun Nov 14 01:00:01 2010] worker1 mydomain.com 271.622400

[Sun Nov 14 01:00:01 2010] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (127.0.0.1:8009), err=-104
[Sun Nov 14 01:00:01 2010] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Sun Nov 14 01:00:06 2010] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (127.0.0.1:8009), err=-104
[Sun Nov 14 01:00:06 2010] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Sun Nov 14 01:00:06 2010] [error] ajp_service::jk_ajp_common.c (1758): Error connecting to tomcat. Tomcat is probably not started or is listening on the wrong port. worker=worker1 failed
[Sun Nov 14 01:00:06 2010] worker1 mydomain.com 86.999946

[Sun Nov 14 01:00:06 2010] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (127.0.0.1:8009), err=-104
[Sun Nov 14 01:00:06 2010] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Sun Nov 14 01:00:10 2010] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (127.0.0.1:8009), err=-104
[Sun Nov 14 01:00:10 2010] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Sun Nov 14 01:00:10 2010] [error] ajp_service::jk_ajp_common.c (1758): Error connecting to tomcat. Tomcat is probably not started or is listening on the wrong port. worker=worker1 failed
[Sun Nov 14 01:00:10 2010] worker1 mydomain.com 39.000187

Meesages in /var/log/messages:

Nov 14 00:59:40 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:00:47 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:01:49 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:02:54 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:04:03 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:05:06 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:06:13 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:07:14 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:08:15 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:09:19 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:10:23 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:11:29 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:12:29 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:13:29 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:14:56 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:16:04 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:17:19 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:18:22 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:19:26 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:20:26 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:21:36 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:22:37 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:23:48 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:24:50 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:25:54 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:26:57 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:29:08 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 01:30:10 nemo kernel: possible SYN flooding on port 8009. Sending cookies.
Nov 14 03:00:01 nemo setroubleshoot: SELinux is preventing mysqld (mysqld_t) "getattr" to /var/lib/mysql/mysql.sock (var_lib_t). For complete SELinux messages. run sea$
Nov 14 03:00:01 nemo setroubleshoot: SELinux is preventing mysqld (mysqld_t) "getattr" to /var/lib/mysql/nemo.me-tech.com.my.err (var_lib_t). For complete SELinux mess$
Nov 14 03:00:01 nemo setroubleshoot: SELinux is preventing mysqld (mysqld_t) "read" to ./time_zone_name.frm (var_lib_t). For complete SELinux messages. run sealert -l $

Apache conf:



# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
 ‹IfModule prefork.c›
StartServers         5
MinSpareServers      5
MaxSpareServers     10
MaxClients          256
MaxRequestsPerChild  0
 ‹/IfModule›

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
 ‹IfModule worker.c›
StartServers         2
MaxClients         256
MinSpareThreads     25
MaxSpareThreads     75
ThreadsPerChild     25
MaxRequestsPerChild  0
 ‹/IfModule›

Apache Mod_jk worker.properties:

 worker.list=worker1
 worker.worker1.type=ajp13
 worker.worker1.host=127.0.0.1
 worker.worker1.port=8009
 worker.worker1.lbfactor=1
 worker.worker1.socket_keepalive=1
 worker.worker1.recycle_timeout=180
 worker.worker1.sticky_session=False

Tomcat server.xml(AJP connector for mod_jk):


   ‹Connector className="org.apache.coyote.tomcat5.CoyoteConnector"
               port="8009"
               protocol="AJP/1.3"
               enableLookups="true"
               redirectPort="8443"
               acceptCount="10"
               debug="0"
               connectionTimeout="5000"
               useURIValidationHack="false"
               maxPostSize="10000000"
               maxThreads="256"
               maxSpareThreads="10"
               URIEncoding="UTF-8"
               protocolHandlerClassName="org.apache.jk.server.JkCoyoteHandler"/›

Any help or advise would be highly appreciated.

Thanks a lot

2
apachetomcat

2 Answers

3
votes

The OOM (Out of memory error) might not necessarily be related to your overall mod_jk/apache configuration, etc. OOMs are typical consequences of memory leaks.

My recommended way of analyzing OOMs is to do a heap dump during the runtime and open it using any Java profiler. Or, even better, add -XX:+HeapDumpOnOutOfMemoryError to your Java parameter list and you will get a dump at the very moment you've got OOM. You might want to add -XX:HeapDumpPath=/some/big/disk/with/space so you know where to look for files.

-1
votes

You can write shell script to restart tomcat once it hit out of memory error.

Edit catalina.sh

Add this -XX:OnOutOfMemoryError=’/your_path/restart_tomcat.sh’ inside your CATALINA_OPTS

Original answer: http://www.droidmsg.com/blog/?p=185