Java OOM in High load project

0
votes

I have problem with OOM Exception in high load multi-thread Java project.
I appreciate any help you can give.

Delails: Project is build on Java+Mysql as storage.

There is no evidence about additional RAM use in moment of application crash(no jumps in any monitoring tool).There is a lot of free memory
CPU and I/O is ok too.
Jstack tool not found any thread deadlocks.
No slow or error logs in Mysql.
In hs_err you can see thread_blocked state but I don't found any reason for this.
Java run args, system limits, free memory you can see below.

Here is error in hs_err_filepart(full file attached):

There is insufficient memory for the Java Runtime Environment to continue. Native memory allocation (mmap) failed to map 1633681408 bytes for committing reserved memory.
Possible reasons:
The system is out of physical RAM or swap space In 32 bit mode, the process size limit was hit
Possible solutions:
Reduce memory load on the system
Increase physical memory or swap space
Check if swap backing store is full
Use 64 bit Java on a 64 bit OS
Decrease Java heap size (-Xmx/-Xms)
Decrease number of Java threads
Decrease Java thread stack sizes (-Xss)
Set larger code cache with -XX:ReservedCodeCacheSize= This output file may be truncated or incomplete.

Out of Memory Error (os_linux.cpp:2627), pid=11980, tid=0x00007f27148c8700

JRE version: Java(TM) SE Runtime Environment (8.0_92-b14) (build 1.8.0_92-b14)
Java VM: Java HotSpot(TM) 64-Bit Server VM (25.92-b14 mixed mode linux-amd64 compressed oops)
Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

--------------- T H R E A D ---------------

Current thread (0x00007f27883c5800): VMThread [stack: 0x00007f27147c8000,0x00007f27148c9000] [id=12005]

Stack: [0x00007f27147c8000,0x00007f27148c9000], sp=0x00007f27148c7160, free space=1020k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xabd65a] VMError::report_and_die()+0x2ba
V [libjvm.so+0x4fb4db] report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*)+0x8b
V [libjvm.so+0x91d713] os::Linux::commit_memory_impl(char*, unsigned long, bool)+0x103
V [libjvm.so+0x91dc69] os::pd_commit_memory(char*, unsigned long, unsigned long, bool)+0x29
V [libjvm.so+0x917f6a] os::commit_memory(char*, unsigned long, unsigned long, bool)+0x2a
V [libjvm.so+0x98c343] PSVirtualSpace::expand_by(unsigned long)+0x53
V [libjvm.so+0x98d748] PSYoungGen::resize_generation(unsigned long, unsigned long)+0xf8
V [libjvm.so+0x98c8a2] PSYoungGen::resize(unsigned long, unsigned long)+0x22
V [libjvm.so+0x989b7b] PSScavenge::invoke_no_policy()+0xf3b
V [libjvm.so+0x98a301] PSScavenge::invoke()+0x41
V [libjvm.so+0x941410] ParallelScavengeHeap::failed_mem_allocate(unsigned long)+0x70
V [libjvm.so+0xabf077] VM_ParallelGCFailedAllocation::doit()+0x97
V [libjvm.so+0xac6aa5] VM_Operation::evaluate()+0x55
V [libjvm.so+0xac4e7a] VMThread::evaluate_operation(VM_Operation*)+0xba
V [libjvm.so+0xac51fe] VMThread::loop()+0x1ce
V [libjvm.so+0xac5670] VMThread::run()+0x70
V [libjvm.so+0x91fad8] java_start(Thread*)+0x108

VM_Operation (0x00007f22963f34e0): ParallelGCFailedAllocation, mode: safepoint, requested by thread 0x00007f25e8105000

--------------- P R O C E S S ---------------

Java Threads: ( => current thread )
0x00007f25a8052800 JavaThread "pool-27-thread-32" [_thread_blocked, id=26278, stack(0x00007f2147523000,0x00007f2147624000)]
0x00007f24dc07f000 JavaThread "pool-33-thread-55" [_thread_blocked, id=26277, stack(0x00007f2147624000,0x00007f2147725000)]
0x00007f25bc06b800 JavaThread "pool-28-thread-48" [_thread_blocked, id=26272, stack(0x00007f2147725000,0x00007f2147826000)]
0x00007f2568060000 JavaThread "pool-24-thread-39" [_thread_blocked, id=26267, stack(0x00007f2147826000,0x00007f2147927000)]
0x00007f25bc06a000 JavaThread "pool-28-thread-47" [_thread_blocked, id=26262, stack(0x00007f2147927000,0x00007f2147a28000)]

jvm_args: -Xmx16G -XX:ReservedCodeCacheSize=256M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/DDT/CPA -XX:-OmitStackTraceInFastThrow

Memory:
free -m
total used free shared buffers cached
Mem: 64298 37022 27275 5 593 9466
-/+ buffers/cache: 26962 37336
Swap: 15847 957 14889

System limits:

ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 256455
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 64000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 256455
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

1
javalinuxmultithreadingout-of-memoryhigh-load
Is it reproducable? Have you considered upgrading from 1.8.0_92-b14 to a more recent version? – Kayaman
Is it reproducible? - No, at test server all OK, even with twice load as current production load. Have you considered upgrading from 1.8.0_92-b14 to a more recent version? - If it will help we of course can do this. But I must prove that upgrade will help. – MisterLotus
You shouldn't have to prove anything (or at least not that much), as updating from 92 to 162 is a micro release. It might be the least effort/most useful attempt, if you don't have any other leads. – Kayaman
This is at least an option, thank you. But i wish to understand why such crash happens. – MisterLotus
It's about the best advice you can hope for. You're running in deep waters here and while there are a lot of smart people here (a lot smarter than me), nobody can effectively remote-debug your system even though you did a fine job providing plenty of information. – Kayaman

1 Answers