I experience prolonged GC pause (> 10 seconds) in Datastax-enterprise's Cassandra/Solr package. After monitoring for days I found it only happens when GC of CMS Perm Gen happens as shown in the chart. And the long GC happens at EVERY flection point of the chart, when PermGen GC happens. And every time when Perm Gen GC kicks in, there's a long pause that cause client sessions timeout!
https://www.dropbox.com/s/qgdcurprvc1sees/permgen_gc.png
Heap GC is normal and has no pauses, only and always get the long pause in the Non-heap Perm Gen GC, which always happens when the server is at non-peak hours.
![enter image description here][1]
JVM options used by DSE:
-ea -javaagent:/usr/local/dse/resources/cassandra/lib/jamm-0.2.5.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42
-Xms16384M -Xmx16384M -Xmn5461M -XX:+HeapDumpOnOutOfMemoryError
-Xss180k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
-XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-Djava.net.preferIPv4Stack=true -Dcassandra.load_ring_state=false
-Dcassandra-foreground=yes -Dsearch-service=true
-Dtomcat.logs=/var/log/dse/tomcat -DName=SI2_DSE
-Ddse.solr.data.dir=/data/solrIndexRamDisk
-Djava.library.path=/usr/local/dse/resources/hadoop/native/Linux-amd64-64/lib
JVM Info
- Virtual Machine: Java HotSpot(TM) 64-Bit Server VM version 20.12-b01
- Vendor: Sun Microsystems Inc.
- JIT compiler: HotSpot 64-Bit Tiered Compilers
Heap Info
- Current heap size: 10,247,153 kbytes
- Maximum heap size: 16,218,048 kbytes
- Committed memory: 16,218,048 kbytes
- Pending finalization: {0} objects
VM Server Info
- Operating System: Linux 2.6.32-279.1.1.el6.x86_64
- Architecture: amd64
- Number of processors: 32
- Committed virtual memory: 39,845,596 kbytes
- Total physical memory: 99,018,824 kbytes
- Free physical memory: 58,184,572 kbytes
- Total swap space: 4,194,296 kbytes
- Free swap space: 4,194,296 kbytes
