3
votes

Our production servers crashed twice with problematic frame inside one of our classes. We have not been able to reproduce this issue and haven't seen it in any other environment. What can cause this?


    #
    # A fatal error has been detected by the Java Runtime Environment:
    #
    #  SIGSEGV (0xb) at pc=0xfbfe2e0a, pid=21857, tid=53
    #
    # JRE version: 6.0_26-b03
    # Java VM: Java HotSpot(TM) Server VM (20.1-b02 mixed mode solaris-x86 )
    # Problematic frame:
    # J  com.xxx.insurance.domain.account.Address.setPostcode(Ljava/lang/String;)V
    #
    # If you would like to submit a bug report, please visit:
    #   http://java.sun.com/webapps/bugreport/crash.jsp
    #

    ---------------  T H R E A D  ---------------

    Current thread (0x08c7b000):  JavaThread "[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_in_Java, id=53, stack(0xa83ee000,0xa843e000)]

    siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x00000010

    Registers:
    EAX=0x00000000, EBX=0xd5c88a20, ECX=0xe5c66910, EDX=0x08c7b000
    ESP=0xa843b1b0, EBP=0x00000000, ESI=0xaaaaf910, EDI=0xe5c66938
    EIP=0xfbfe2e0a, EFLAGS=0x00010282

    Top of Stack: (sp=0xa843b1b0)
    0xa843b1b0:   00000000 aaaaf910 e5c66938 d5c88a20
    0xa843b1c0:   e5c66910 b0b33f98 e5c66938 aab7a360
    0xa843b1d0:   00000000 a843b1e4 a843b20c fae0310d
    0xa843b1e0:   fae0310d 00000000 e5c66910 a843b1ec
    0xa843b1f0:   b47d8dc8 a843b21c b47d9000 b47e7e48
    0xa843b200:   b47d8de8 a843b1e4 a843b220 e5c66a68
    0xa843b210:   fb59c440 e5c66a68 e5c66910 d5b39ac0
    0xa843b220:   e5c66a68 fddaaf00 00000000 e5c66910 

    Instructions: (pc=0xfbfe2e0a)
    0xfbfe2dea:   7c b4 8b 4f 14 3b f5 0f 84 41 02 00 00 89 4c 24
    0xfbfe2dfa:   10 89 5c 24 0c 89 7c 24 08 89 74 24 04 89 2c 24
    0xfbfe2e0a:   8b 6d 10 85 f6 0f 84 cd 00 00 00 8b 0c 24 85 c9
    0xfbfe2e1a:   0f 84 c2 00 00 00 8b 4e 10 3b cd 0f 85 b7 00 00 

    Register to memory mapping:

    EAX=0x00000000 is an unknown value
    EBX=0xd5c88a20 is an oop
    com.xxx.insurance.domain.account.Address$Property 
     - klass: 'com/xxx/insurance/domain/account/Address$Property'
    ECX=0xe5c66910 is an oop
    com.xxx.insurance.domain.account.Address 
     - klass: 'com/xxx/insurance/domain/account/Address'
    EDX=0x08c7b000 is a thread
    ESP=0xa843b1b0 is pointing into the stack for thread: 0x08c7b000
    EBP=0x00000000 is an unknown value
    ESI=0xaaaaf910 is an oop
    java.lang.String 
     - klass: 'java/lang/String'
    EDI=0xe5c66938 is an oop
    com.xxx.insurance.beans.InsurancePropertyChangeSupport 
     - klass: 'com/xxx/insurance/beans/InsurancePropertyChangeSupport'


    Stack: [0xa83ee000,0xa843e000],  sp=0xa843b1b0,  free space=308k
    Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
    J  com.xxx.insurance.domain.account.Address.setPostcode(Ljava/lang/String;)V

    [error occurred during error reporting (printing native stack), id 0xb]


    ---------------  P R O C E S S  ---------------

    Java Threads: ( => current thread )
      0x0bbb8c00 JavaThread "Keep-Alive-Timer" daemon [_thread_blocked, id=371, stack(0xa7bff000,0xa7c4f000)]
      0x0c65f800 JavaThread "RHTTP downloader (general)" daemon [_thread_in_native, id=360, stack(0xa6b84000,0xa6bd4000)]
      0x0d431400 JavaThread "RangeSocketListener-1" daemon [_thread_blocked, id=353, stack(0xa6c26000,0xa6c76000)]
      0x0b397000 JavaThread "RHTTP downloader (general)" daemon [_thread_blocked, id=333, stack(0xa6c77000,0xa6cc7000)]
      0x0a7df800 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=275, stack(0xa771d000,0xa776d000)]
      0x0d198400 JavaThread "Store defaultMethodCache Spool Thread" daemon [_thread_blocked, id=218, stack(0xa7662000,0xa76b2000)]
      0x0c562800 JavaThread "[STANDBY] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked, id=78, stack(0xa76b3000,0xa7703000)]
      0x09b57800 JavaThread "DynamicSSLListenThread[DefaultSecure]" daemon [_thread_in_native, id=76, stack(0xa779c000,0xa77ec000)]
      0x0824ac00 JavaThread "weblogic.GCMonitor" daemon [_thread_blocked, id=75, stack(0xa77ed000,0xa783d000)]
      0x0acdd400 JavaThread "[STANDBY] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked, id=74, stack(0xa783e000,0xa788e000)]
      0x08248c00 JavaThread "[STANDBY] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked, id=73, stack(0xa788f000,0xa78df000)]
      0x0ca4c400 JavaThread "weblogic.cluster.MessageReceiver" daemon [_thread_in_native, id=71, stack(0xa790e000,0xa795e000)]
      0x08249400 JavaThread "Thread-19" [_thread_blocked, id=70, stack(0xa795f000,0xa79af000)]
      0x0bc1cc00 JavaThread "Store defaultMethodCache Spool Thread" daemon [_thread_blocked, id=69, stack(0xa7abb000,0xa7b0b000)]
      0x0b7b8000 JavaThread "jboss EL reference queue cleanup thread" daemon [_thread_blocked, id=68, stack(0xa7b0c000,0xa7b5c000)]
      0x0855f000 JavaThread "[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked, id=64, stack(0xa80e7000,0xa8137000)]
      0x0a659400 JavaThread "Timer-3" daemon [_thread_blocked, id=63, stack(0xa8c47000,0xa8c97000)]
      0x0aa22800 JavaThread "Thread-16" [_thread_blocked, id=62, stack(0xa8189000,0xa81d9000)]
      0x0a776c00 JavaThread "DoSManager" daemon [_thread_blocked, id=61, stack(0xa8138000,0xa8188000)]
      0x0acd6000 JavaThread "VDE Transaction Processor Thread" daemon [_thread_blocked, id=59, stack(0xa81da000,0xa822a000)]
      0x0ad9ec00 JavaThread "Timer-2" daemon [_thread_blocked, id=58, stack(0xa822b000,0xa827b000)]
      0x0a40ec00 JavaThread "ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'" daemon [_thread_blocked, id=57, stack(0xa827c000,0xa82cc000)]
      0x086b3400 JavaThread "ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" daemon [_thread_blocked, id=56, stack(0xa82cd000,0xa831d000)]
      0x09598000 JavaThread "ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" daemon [_thread_blocked, id=55, stack(0xa831e000,0xa836e000)]
      0x0d53b800 JavaThread "ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" daemon [_thread_in_native, id=54, stack(0xa836f000,0xa83bf000)]
    =>0x08c7b000 JavaThread "[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_in_Java, id=53, stack(0xa83ee000,0xa843e000)]
      0x0a31c000 JavaThread "NM Reaper" daemon [_thread_blocked, id=52, stack(0xa843f000,0xa848f000)]
      0x0a5d4400 JavaThread "Timer-1" daemon [_thread_blocked, id=51, stack(0xa84ac000,0xa84fc000)]
      0x0a319800 JavaThread "weblogic.timers.TimerThread" daemon [_thread_blocked, id=50, stack(0xa84fd000,0xa854d000)]
      0x0a763400 JavaThread "weblogic.time.TimeEventGenerator" daemon [_thread_blocked, id=49, stack(0xa854e000,0xa859e000)]
      0x0a3b2000 JavaThread "[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked, id=48, stack(0xa859f000,0xa85ef000)]
      0x0a703c00 JavaThread "Timer-0" daemon [_thread_blocked, id=47, stack(0xa8bbf000,0xa8c0f000)]
      0x08a73400 JavaThread "RHTTP Client for 'insurRhodesServer3:35006'" daemon [_thread_blocked, id=40, stack(0xa9a8f000,0xa9adf000)]
      0x08f44000 JavaThread "Acceptor ServerSocket[addr=/0.0.0.0,port=0,localport=35006]" daemon [_thread_in_native, id=38, stack(0xa8ed3000,0xa8f23000)]
      0x08e2b000 JavaThread "SessionScavenger" daemon [_thread_blocked, id=37, stack(0xa8f4e000,0xa8f9e000)]
      0x08ccd800 JavaThread "SessionScavenger" daemon [_thread_blocked, id=36, stack(0xa9aff000,0xa9b4f000)]
      0x08a70000 JavaThread "SessionScavenger" daemon [_thread_blocked, id=35, stack(0xa9d06000,0xa9d56000)]
      0x08437c00 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=32, stack(0xa97fd000,0xa984d000)]
      0x08438400 JavaThread "C2 CompilerThread1" daemon [_thread_in_native, id=31, stack(0xa984e000,0xa98ce000)]
      0x08436000 JavaThread "C2 CompilerThread0" daemon [_thread_in_native, id=30, stack(0xa98cf000,0xa994f000)]
      0x0842ac00 JavaThread "Metrics Collection" daemon [_thread_blocked, id=29, stack(0xa997e000,0xa99ce000)]
      0x08425000 JavaThread "Stack Trace Sampler" daemon [_thread_blocked, id=28, stack(0xa99cf000,0xa9a1f000)]
      0x0841d800 JavaThread "ServerCommunication" daemon [_thread_blocked, id=27, stack(0xa9a3e000,0xa9a8e000)]
      0x083b6400 JavaThread "Buffer flushing thread" daemon [_thread_blocked, id=23, stack(0xa9c13000,0xa9c63000)]
      0x083d3400 JavaThread "metric-sampler" daemon [_thread_blocked, id=22, stack(0xa9c64000,0xa9cb4000)]
      0x0838f800 JavaThread "shared InfrequentEventScheduler" daemon [_thread_blocked, id=21, stack(0xa9cb5000,0xa9d05000)]
      0x081f9c00 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=19, stack(0xa9e0a000,0xa9e5a000)]
      0x081e8c00 JavaThread "Finalizer" daemon [_thread_blocked, id=18, stack(0xaa8ad000,0xaa8fd000)]
      0x081e4000 JavaThread "Reference Handler" daemon [_thread_blocked, id=17, stack(0xfde2e000,0xfde7e000)]
      0x08072c00 JavaThread "main" [_thread_blocked, id=2, stack(0xfe08f000,0xfe0df000)]

    Other Threads:
      0x081e0800 VMThread [stack: 0xa9e5b000,0xa9edb000] [id=16]
      0x08432800 WatcherThread [stack: 0xa977c000,0xa97fc000] [id=33]

    VM state:not at safepoint (normal execution)

    VM Mutex/Monitor currently owned by a thread: None

    Heap
     PSYoungGen      total 332096K, used 16671K [0xe5600000, 0xfac00000, 0xfac00000)
      eden space 314304K, 2% used [0xe5600000,0xe5d77580,0xf88f0000)
      from space 17792K, 50% used [0xf88f0000,0xf91c0988,0xf9a50000)
      to   space 17216K, 0% used [0xf9b30000,0xf9b30000,0xfac00000)
     PSOldGen        total 700416K, used 523951K [0xbaa00000, 0xe5600000, 0xe5600000)
      object space 700416K, 74% used [0xbaa00000,0xda9abd00,0xe5600000)
     PSPermGen       total 262144K, used 187475K [0xaaa00000, 0xbaa00000, 0xbaa00000)
      object space 262144K, 71% used [0xaaa00000,0xb6114fd8,0xbaa00000)

    Code Cache  [0xfae00000, 0xfbfe8000, 0xfde00000)
     total_blobs=5279 nmethods=4926 adapters=305 free_code_cache=31627328 largest_free_block=6016

2
Since it could be a bug in the JVM, the first thing I would suggest is trying the latest version of Java 6 update 31.Peter Lawrey
And if you are seeing this in 2017 ... upgrade to Java 8!Stephen C

2 Answers

2
votes

What can cause this?

The cause is likely to be either a bug in some native code library that your application is using, or a JVM bug.

In the former case, you'll need to brush up on your native code debugging skills. (You know, using gdb and its ilk.)

In the latter case, upgrading to a more recent JVM patch level may help. Or you could try scanning the "bugs fixed" sections of the release notes for later JVM releases. Or you could trawl through Oracle's Java Bug Database (though no promising search terms leap out at me ...)

0
votes

If the native libraries are involved, it is worth checking DLL/Shared Libraries of production system against recommended patch levels.