I'm trying to avoid the Full GC (from gc.log sample below) running a Grails application in Tomcat in production. Any suggestions on how to better configure the GC?
14359.317: [Full GC 14359.317: [CMS: 3453285K->3099828K(4194304K), 13.1778420 secs] 4506618K->3099828K(6081792K), [CMS Perm : 261951K->181304K(264372K)] icms_dc=0 , 13.1786310 secs] [Times: user=13.15 sys=0.04, real=13.18 secs]
My VM params are as follow:
-Xms=6G
-Xmx=6G
-XX:MaxPermSize=1G
-XX:NewSize=2G
-XX:MaxTenuringThreshold=8
-XX:SurvivorRatio=7
-XX:+UseConcMarkSweepGC
-XX:+CMSClassUnloadingEnabled
-XX:+CMSPermGenSweepingEnabled
-XX:+CMSIncrementalMode
-XX:CMSInitiatingOccupancyFraction=60
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+HeapDumpOnOutOfMemoryError
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintTenuringDistribution
-Dsun.reflect.inflationThreshold=0
14169.764: [GC 14169.764: [ParNew Desired survivor size 107347968 bytes, new threshold 8 (max 8) - age 1: 15584312 bytes, 15584312 total - age 2: 20053704 bytes, 35638016 total - age 3: 13624872 bytes, 49262888 total - age 4: 14469608 bytes, 63732496 total - age 5: 10553288 bytes, 74285784 total - age 6: 11797648 bytes, 86083432 total - age 7: 12591328 bytes, 98674760 total : 1826161K->130133K(1887488K), 0.1726640 secs] 5216326K->3537160K(6081792K) icms_dc=0 , 0.1733010 secs] [Times: user=0.66 sys=0.03, real=0.17 secs] 14218.712: [GC 14218.712: [ParNew Desired survivor size 107347968 bytes, new threshold 8 (max 8) - age 1: 25898512 bytes, 25898512 total - age 2: 10308160 bytes, 36206672 total - age 3: 16927792 bytes, 53134464 total - age 4: 13493608 bytes, 66628072 total - age 5: 14301832 bytes, 80929904 total - age 6: 10448408 bytes, 91378312 total - age 7: 11724056 bytes, 103102368 total - age 8: 12299528 bytes, 115401896 total : 1807957K->147911K(1887488K), 0.1664510 secs] 5214984K->3554938K(6081792K) icms_dc=0 , 0.1671290 secs] [Times: user=0.61 sys=0.00, real=0.17 secs] 14251.429: [GC 14251.430: [ParNew Desired survivor size 107347968 bytes, new threshold 7 (max 8) - age 1: 25749296 bytes, 25749296 total - age 2: 20111888 bytes, 45861184 total - age 3: 7580776 bytes, 53441960 total - age 4: 16819072 bytes, 70261032 total - age 5: 13209968 bytes, 83471000 total - age 6: 14088856 bytes, 97559856 total - age 7: 10371160 bytes, 107931016 total - age 8: 11426712 bytes, 119357728 total : 1825735K->155304K(1887488K), 0.1888880 secs] 5232762K->3574222K(6081792K) icms_dc=0 , 0.1895340 secs] [Times: user=0.74 sys=0.06, real=0.19 secs] 14291.342: [GC 14291.343: [ParNew Desired survivor size 107347968 bytes, new threshold 7 (max 8) - age 1: 25786480 bytes, 25786480 total - age 2: 21991848 bytes, 47778328 total - age 3: 16650000 bytes, 64428328 total - age 4: 7387368 bytes, 71815696 total - age 5: 16777584 bytes, 88593280 total - age 6: 13098856 bytes, 101692136 total - age 7: 14029704 bytes, 115721840 total : 1833128K->151603K(1887488K), 0.1941170 secs] 5252046K->3591384K(6081792K) icms_dc=0 , 0.1947390 secs] [Times: user=0.82 sys=0.04, real=0.20 secs] 14334.142: [GC 14334.143: [ParNew Desired survivor size 107347968 bytes, new threshold 6 (max 8) - age 1: 31541800 bytes, 31541800 total - age 2: 20826888 bytes, 52368688 total - age 3: 19155264 bytes, 71523952 total - age 4: 16422240 bytes, 87946192 total - age 5: 7235616 bytes, 95181808 total - age 6: 16549000 bytes, 111730808 total - age 7: 13026064 bytes, 124756872 total : 1829427K->167467K(1887488K), 0.1890190 secs] 5269208K->3620753K(6081792K) icms_dc=0 , 0.1896630 secs] [Times: user=0.80 sys=0.03, real=0.19 secs] 14359.317: [Full GC 14359.317: [CMS: 3453285K->3099828K(4194304K), 13.1778420 secs] 4506618K->3099828K(6081792K), [CMS Perm : 261951K->181304K(264372K)] icms_dc=0 , 13.1786310 secs] [Times: user=13.15 sys=0.04, real=13.18 secs] 14373.287: [GC [1 CMS-initial-mark: 3099828K(4194304K)] 3100094K(6081792K), 0.0107380 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 14373.298: [CMS-concurrent-mark-start] 14472.579: [GC 14472.579: [ParNew Desired survivor size 107347968 bytes, new threshold 8 (max 8) - age 1: 42849392 bytes, 42849392 total : 1677824K->86719K(1887488K), 0.1056680 secs] 4777652K->3186547K(6081792K) icms_dc=0 , 0.1063280 secs] [Times: user=0.61 sys=0.00, real=0.11 secs] 14506.980: [GC 14506.980: [ParNew Desired survivor size 107347968 bytes, new threshold 8 (max 8) - age 1: 42002904 bytes, 42002904 total - age 2: 35733928 bytes, 77736832 total : 1764543K->96136K(1887488K), 0.0982790 secs] 4864371K->3195964K(6081792K) icms_dc=0 , 0.0988960 secs] [Times: user=0.53 sys=0.01, real=0.10 secs] 14544.285: [GC 14544.286: [ParNew Desired survivor size 107347968 bytes, new threshold 8 (max 8) - age 1: 26159736 bytes, 26159736 total - age 2: 37842840 bytes, 64002576 total - age 3: 33192784 bytes, 97195360 total : 1773960K->130799K(1887488K), 0.1208590 secs] 4873788K->3230628K(6081792K) icms_dc=0 , 0.1215900 secs] [Times: user=0.59 sys=0.02, real=0.13 secs] 14589.266: [GC 14589.266: [ParNew Desired survivor size 107347968 bytes, new threshold 4 (max 8) - age 1: 28010360 bytes, 28010360 total - age 2: 21136704 bytes, 49147064 total - age 3: 35081376 bytes, 84228440 total - age 4: 32468056 bytes, 116696496 total : 1808623K->148284K(1887488K), 0.1423150 secs] 4908452K->3248112K(6081792K) icms_dc=0 , 0.1429440 secs] [Times: user=0.70 sys=0.02, real=0.14 secs] 14630.947: [GC 14630.947: [ParNew Desired survivor size 107347968 bytes, new threshold 8 (max 8) - age 1: 28248240 bytes, 28248240 total - age 2: 20712320 bytes, 48960560 total - age 3: 18217168 bytes, 67177728 total - age 4: 34834832 bytes, 102012560 total : 1826108K->140347K(1887488K), 0.1784680 secs] 4925936K->3275469K(6081792K) icms_dc=0 , 0.1790920 secs] [Times: user=0.98 sys=0.03, real=0.18 secs] 14664.779: [GC 14664.779: [ParNew Desired survivor size 107347968 bytes, new threshold 5 (max 8) - age 1: 25841000 bytes, 25841000 total - age 2: 22264960 bytes, 48105960 total - age 3: 17730104 bytes, 65836064 total - age 4: 17988048 bytes, 83824112 total - age 5: 34739384 bytes, 118563496 total : 1818171K->147603K(1887488K), 0.1714160 secs] 4953293K->3282725K(6081792K) icms_dc=0 , 0.1720530 secs] [Times: user=0.82 sys=0.11, real=0.17 secs] 14702.488: [GC 14702.489: [ParNew Desired survivor size 107347968 bytes, new threshold 8 (max 8) - age 1: 26887368 bytes, 26887368 total - age 2: 21403352 bytes, 48290720 total - age 3: 18732224 bytes, 67022944 total - age 4: 17640576 bytes, 84663520 total - age 5: 17942952 bytes, 102606472 total : 1825427K->142695K(1887488K), 0.2118320 secs] 4960549K->3312168K(6081792K) icms_dc=0 , 0.2124630 secs] [Times: user=1.13 sys=0.14, real=0.21 secs]
The strategy I was aiming at: I want to limit to the minimum what gets Tenured, I'm serving requests and expect that beyond a certain amount of shared objects, every other objects are useful only to the request at hand. Therefore by using a big NewSize and an increased TenuringThreshold and was hoping to have none of these single serving objects stick around.
The following are there to support my strategy:
-Xms=6G
-Xmx=6G
-XX:NewSize=2G // big space so that ParNew doesn't occur to often and let time for objects to expire
-XX:MaxTenuringThreshold=8 // to limit the tenuring some more
-XX:SurvivorRatio=7 // based on examples
-XX:CMSInitiatingOccupancyFraction=60
// to prevent a Full GC caused by promotion allocation failed
-XX:+UseCMSInitiatingOccupancyOnly
// to go with the one above based on example
MaxPermSize=1G and "-Dsun.reflect.inflationThreshold=0" are related to another issue I'd rather keep separated.
"-XX:+CMSClassUnloadingEnabled" and "-XX:+CMSPermGenSweepingEnabled" are there because of grails which rely heavily and extra classes for closures and reflexion
-XX:+CMSIncrementalMode is an experiment which hasn't yield much success