I'm using Spring boot for the application. There is a particular controller when called, makes a couple of independent requests using OkHttpClient. I am using a thread pool of 200 threads to make these calls. It works absolutely fine in the local environment. However, when deployed to cloud foundry, it shows up an error as follows.
I have reduced the number of threads to 100 but the problem still remains. These are the memory settings in my CF instance -
Instance Memory Limit 8192 MB
Memory Quota 3072 MB (available memory 1024 MB)
Disk Quota 1024 MB
This is the error -
2019-05-08T19:13:35.68+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:35.686 INFO 6 --- [ool-2-thread-65] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:35.76+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:35.769 INFO 6 --- [ool-2-thread-62] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:35.81+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:35.813 INFO 6 --- [ool-2-thread-54] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:35.95+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:35.959 INFO 6 --- [ool-2-thread-73] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:35.96+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:35.967 INFO 6 --- [ool-2-thread-60] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:35.96+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:35.967 INFO 6 --- [ool-2-thread-79] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.03+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.034 INFO 6 --- [ool-2-thread-77] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.05+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.054 INFO 6 --- [ool-2-thread-75] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.06+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.066 INFO 6 --- [ool-2-thread-18] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.11+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.119 INFO 6 --- [ool-2-thread-12] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.12+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.121 INFO 6 --- [ool-2-thread-32] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.12+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.123 INFO 6 --- [ool-2-thread-71] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.12+0530 [APP/PROC/WEB/0] ERR Resource exhaustion event: the JVM was unable to create a thread.
2019-05-08T19:13:36.12+0530 [APP/PROC/WEB/0] ERR ResourceExhausted! (1/0)
2019-05-08T19:13:36.13+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.136 INFO 6 --- [ool-2-thread-44] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT | Instance Count | Total Bytes | Class Name |
2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.583 INFO 6 --- [ool-2-thread-92] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.583 INFO 6 --- [ool-2-thread-59] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.583 INFO 6 --- [ool-2-thread-38] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.584 INFO 6 --- [ool-2-thread-25] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT | 100796 | 11914496 | [C |
2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.584 INFO 6 --- [ool-2-thread-78] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.585 INFO 6 --- [ool-2-thread-67] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.585 INFO 6 --- [ol-2-thread-100] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT | 99630 | 2391120 | Ljava/lang/String; |
2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.585 INFO 6 --- [ool-2-thread-80] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.586 INFO 6 --- [ool-2-thread-10] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT | 6940 | 2320424 | [B |
2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT | 41544 | 1661760 | Ljava/util/LinkedHashMap$Entry; |
2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT | 50549 | 1617568 | Ljava/util/HashMap$Node; |
2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 13832 | 1535128 | Ljava/lang/Class; |
2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 42807 | 1369824 | Ljava/util/concurrent/ConcurrentHashMap$Node; |
2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 22775 | 1279152 | [Ljava/util/HashMap$Node; |
2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 21141 | 1183896 | Ljava/util/LinkedHashMap; |
2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 10387 | 914056 | Ljava/lang/reflect/Method; |
2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 11925 | 913696 | [Ljava/lang/Object; |
2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 2202 | 618392 | [I |
2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 36548 | 584768 | Ljava/lang/Object; |
2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 1168 | 485888 | Lcom/sap/LMChatbot/services/metricSearch; |
2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 992 | 372992 | Ljava/lang/Thread; |
2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 200 | 370440 | [Ljava/util/concurrent/ConcurrentHashMap$Node; |
2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 18205 | 291280 | Ljava/util/LinkedHashSet; |
2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 1 | 262160 | [Lorg/h2/mvstore/MVMap; |
2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 8144 | 195456 | Ljava/util/ArrayList; |
2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 2277 | 182544 | [Ljava/lang/ThreadLocal$ThreadLocalMap$Entry; |
2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 7532 | 172640 | [Ljava/lang/Class; |
2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 2964 | 142272 | Lorg/json/simple/JSONObject; |
2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 2177 | 121576 | [Ljava/security/ProtectionDomain; |
2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 2406 | 114776 | [Ljava/lang/String; |
2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 4127 | 99048 | Lorg/springframework/core/MethodClassKey; |
2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 1986 | 95328 | Ljava/util/HashMap; |
2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 845 | 94640 | Ljava/net/SocksSocketImpl; |
2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 2319 | 92760 | Ljava/util/TreeMap$Entry; |
2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 1648 | 92288 | Lokio/Okio$4; |
2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 2185 | 87400 | Ljava/security/AccessControlContext; |
2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 3042 | 73008 | Ljava/util/Collections$UnmodifiableRandomAccessList; |
2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 2242 | 71744 | Ljava/util/LinkedList; |
2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT | 2612 | 62688 | Ljava/util/LinkedList$Node; |
2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT | 826 | 59472 | Lokhttp3/internal/connection/RealConnection; |
2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT | 1846 | 59072 | Ljava/util/concurrent/locks/ReentrantLock$NonfairSync; |
2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT | 801 | 57672 | Lorg/springframework/core/annotation/AnnotationAttributes; |
2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT | 1778 | 56896 | Ljava/net/InetAddress$InetAddressHolder; |
2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.625 INFO 6 --- [ool-2-thread-88] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.625 INFO 6 --- [ool-2-thread-48] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.626 INFO 6 --- [ool-2-thread-64] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.626 INFO 6 --- [ool-2-thread-91] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT | 1417 | 56680 | Ljava/lang/invoke/MethodType; |
2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.626 INFO 6 --- [ool-2-thread-43] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.627 INFO 6 --- [ool-2-thread-86] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.627 INFO 6 --- [ool-2-thread-39] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT | 2277 | 54648 | Ljava/lang/ThreadLocal$ThreadLocalMap; |
2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT | 1671 | 53472 | Ljava/lang/ref/ReferenceQueue; |
2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT | 943 | 52808 | Ljava/lang/Package; |
2019-05-08T19:13:36.63+0530 [APP/PROC/WEB/0] OUT | 1059 | 50832 | Lorg/apache/tomcat/util/modeler/AttributeInfo; |
2019-05-08T19:13:36.63+0530 [APP/PROC/WEB/0] OUT | 884 | 49504 | Lokhttp3/Address; |
2019-05-08T19:13:36.63+0530 [APP/PROC/WEB/0] OUT | 1505 | 48160 | Ljava/util/Hashtable$Entry; |
2019-05-08T19:13:36.63+0530 [APP/PROC/WEB/0] OUT | 998 | 47904 | Ljava/util/concurrent/ThreadPoolExecutor$Worker; |
2019-05-08T19:13:36.63+0530 [APP/PROC/WEB/0] OUT | 984 | 47232 | Lokhttp3/HttpUrl; |
2019-05-08T19:13:36.63+0530 [APP/PROC/WEB/0] OUT | 1419 | 45408 | Ljava/lang/invoke/MethodType$ConcurrentWeakInternSet$WeakEntry; |
2019-05-08T19:13:36.63+0530 [APP/PROC/WEB/0] OUT | 1778 | 42672 | Ljava/net/Inet4Address; |
2019-05-08T19:13:36.63+0530 [APP/PROC/WEB/0] OUT | 1776 | 42624 | Ljava/net/InetSocketAddress$InetSocketAddressHolder; |
2019-05-08T19:13:36.64+0530 [APP/PROC/WEB/0] OUT | 1331 | 42592 | Ljava/util/concurrent/FutureTask; |
2019-05-08T19:13:36.64+0530 [APP/PROC/WEB/0] OUT | 866 | 41568 | Lorg/springframework/core/ResolvableType; |
2019-05-08T19:13:36.64+0530 [APP/PROC/WEB/0] OUT | 10 | 41120 | [Ljava/nio/ByteBuffer; |
2019-05-08T19:13:36.64+0530 [APP/PROC/WEB/0] OUT | 841 | 40368 | Ljava/net/SocketOutputStream; |
2019-05-08T19:13:36.64+0530 [APP/PROC/WEB/0] OUT | 841 | 40368 | Ljava/net/SocketInputStream; |
2019-05-08T19:13:36.64+0530 [APP/PROC/WEB/0] OUT | 997 | 39880 | Ljava/lang/ref/Finalizer; |
2019-05-08T19:13:36.64+0530 [APP/PROC/WEB/0] OUT | 1649 | 39576 | Lokio/Buffer; |
2019-05-08T19:13:36.64+0530 [APP/PROC/WEB/0] OUT | 819 | 39312 | Lch/qos/logback/classic/Logger; |
2019-05-08T19:13:36.64+0530 [APP/PROC/WEB/0] OUT | 960 | 38400 | Lorg/springframework/util/ConcurrentReferenceHashMap$Segment; |
2019-05-08T19:13:36.65+0530 [APP/PROC/WEB/0] OUT | 960 | 36544 | [Lorg/springframework/util/ConcurrentReferenceHashMap$Reference; |
2019-05-08T19:13:36.65+0530 [APP/PROC/WEB/0] OUT | 401 | 36336 | [Ljava/util/Hashtable$Entry; |
2019-05-08T19:13:36.65+0530 [APP/PROC/WEB/0] OUT | 1135 | 36320 | Lsun/security/util/DerInputBuffer; |
2019-05-08T19:13:36.65+0530 [APP/PROC/WEB/0] OUT | 1135 | 36320 | Lsun/security/util/DerValue; |
2019-05-08T19:13:36.65+0530 [APP/PROC/WEB/0] OUT | 560 | 35840 | Ljava/util/concurrent/ConcurrentHashMap; |
2019-05-08T19:13:36.65+0530 [APP/PROC/WEB/0] OUT | 743 | 35664 | Lorg/springframework/util/ConcurrentReferenceHashMap$SoftEntryReference; |
2019-05-08T19:13:36.65+0530 [APP/PROC/WEB/0] OUT | 887 | 35480 | Ljava/math/BigInteger; |
2019-05-08T19:13:36.65+0530 [APP/PROC/WEB/0] OUT | 886 | 35440 | Lokhttp3/ConnectionPool; |
2019-05-08T19:13:36.65+0530 [APP/PROC/WEB/0] OUT | 1413 | 33912 | Ljava/util/jar/Attributes$Name; |
2019-05-08T19:13:36.66+0530 [APP/PROC/WEB/0] OUT | 134 | 33712 | [[C |
2019-05-08T19:13:36.66+0530 [APP/PROC/WEB/0] OUT | 1351 | 32424 | Lorg/json/simple/JSONArray; |
2019-05-08T19:13:36.66+0530 [APP/PROC/WEB/0] OUT | 184 | 32384 | Lorg/springframework/context/annotation/ConfigurationClassBeanDefinitionReader$ConfigurationClassBeanDefinition; |
2019-05-08T19:13:36.66+0530 [APP/PROC/WEB/0] OUT | 1295 | 31080 | Ljava/util/ArrayDeque; |
2019-05-08T19:13:36.66+0530 [APP/PROC/WEB/0] OUT | 904 | 28928 | Ljava/io/FileDescriptor; |
2019-05-08T19:13:36.66+0530 [APP/PROC/WEB/0] OUT | 1776 | 28416 | Ljava/net/InetSocketAddress; |
2019-05-08T19:13:36.66+0530 [APP/PROC/WEB/0] OUT | 580 | 27840 | Ljava/util/concurrent/locks/ReentrantReadWriteLock$NonfairSync; |
2019-05-08T19:13:36.66+0530 [APP/PROC/WEB/0] OUT | 312 | 27456 | Lorg/h2/table/Column; |
2019-05-08T19:13:36.66+0530 [APP/PROC/WEB/0] OUT | 1135 | 27240 | Lsun/security/util/DerInputStream; |
2019-05-08T19:13:36.67+0530 [APP/PROC/WEB/0] OUT | 841 | 26912 | Ljava/net/Socket; |
2019-05-08T19:13:36.67+0530 [APP/PROC/WEB/0] OUT | 1675 | 26800 | Ljava/lang/ref/ReferenceQueue$Lock; |
2019-05-08T19:13:36.67+0530 [APP/PROC/WEB/0] OUT | 1114 | 26736 | Lsun/security/x509/AVA; |
2019-05-08T19:13:36.67+0530 [APP/PROC/WEB/0] OUT | 1114 | 26736 | [Lsun/security/x509/AVA; |
2019-05-08T19:13:36.67+0530 [APP/PROC/WEB/0] OUT | 1114 | 26736 | Lsun/security/x509/RDN; |
2019-05-08T19:13:36.67+0530 [APP/PROC/WEB/0] OUT | 1075 | 25800 | Ljava/util/concurrent/LinkedBlockingQueue$Node; |
2019-05-08T19:13:36.67+0530 [APP/PROC/WEB/0] OUT | 447 | 25032 | Ljava/lang/invoke/MemberName; |
2019-05-08T19:13:36.67+0530 [APP/PROC/WEB/0] OUT | 598 | 23920 | Lokhttp3/internal/http/RealInterceptorChain; |
2019-05-08T19:13:36.67+0530 [APP/PROC/WEB/0] OUT | 297 | 23760 | Ljava/lang/reflect/Constructor; |
2019-05-08T19:13:36.68+0530 [APP/PROC/WEB/0] OUT | 140 | 23520 | Lorg/springframework/beans/factory/support/RootBeanDefinition; |
2019-05-08T19:13:36.68+0530 [APP/PROC/WEB/0] OUT | 1443 | 23088 | Ljava/util/LinkedHashMap$LinkedKeySet; |
2019-05-08T19:13:36.68+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.682 INFO 6 --- [ool-2-thread-83] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.68+0530 [APP/PROC/WEB/0] OUT | 960 | 23040 | Lorg/springframework/util/ConcurrentReferenceHashMap$ReferenceManager; |
2019-05-08T19:13:36.68+0530 [APP/PROC/WEB/0] OUT | 477 | 22896 | Lorg/apache/tomcat/util/modeler/OperationInfo; |
2019-05-08T19:13:36.68+0530 [APP/PROC/WEB/0] OUT | 290 | 22744 | [Z |
2019-05-08T19:13:36.68+0530 [APP/PROC/WEB/0] OUT | 566 | 22640 | Ljava/lang/ref/SoftReference; |
2019-05-08T19:13:36.68+0530 [APP/PROC/WEB/0] OUT | 1332 | 21312 | Lorg/springframework/core/annotation/AnnotationUtils$DefaultValueHolder; |
2019-05-08T19:13:36.68+0530 [APP/PROC/WEB/0] OUT | 884 | 21216 | Lokhttp3/Route; |
2019-05-08T19:13:36.69+0530 [APP/PROC/WEB/0] OUT | 882 | 21168 | Ljava/net/Proxy; |
2019-05-08T19:13:36.69+0530 [APP/PROC/WEB/0] OUT | 326 | 20864 | Ljava/net/URL; |
2019-05-08T19:13:36.69+0530 [APP/PROC/WEB/0] OUT | 630 | 20160 | Ljavax/management/MBeanAttributeInfo; |
2019-05-08T19:13:36.69+0530 [APP/PROC/WEB/0] OUT | 824 | 19776 | Lokio/RealBufferedSource; |
2019-05-08T19:13:36.69+0530 [APP/PROC/WEB/0] OUT | 824 | 19776 | Lokio/Okio$2; |
2019-05-08T19:13:36.69+0530 [APP/PROC/WEB/0] OUT | 824 | 19776 | Lokio/Okio$1; |
2019-05-08T19:13:36.69+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.695 INFO 6 --- [ool-2-thread-56] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.69+0530 [APP/PROC/WEB/0] OUT | 824 | 19776 | Lokio/RealBufferedSink; |
2019-05-08T19:13:36.69+0530 [APP/PROC/WEB/0] ERR ERROR: PoolStats action failed: cannot determine memory usage statistics since the JVM is unable to create a thread
2019-05-08T19:13:36.69+0530 [APP/PROC/WEB/0] ERR jvmkill killing current process
2019-05-08T19:13:36.70+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36
2019-05-08T19:13:36.70+0530 [APP/PROC/WEB/0] OUT Full thread dump OpenJDK 64-Bit Server VM (25.212-b03 mixed mode):
2019-05-08T19:13:36.70+0530 [APP/PROC/WEB/0] OUT "OkHttp ConnectionPool" #2406 daemon prio=5 os_prio=0 tid=0x00005641e28d4000 nid=0x467 in Object.wait() [0x00007ff04cd4f000]
2019-05-08T19:13:36.70+0530 [APP/PROC/WEB/0] OUT java.lang.Thread.State: TIMED_WAITING (on object monitor)
2019-05-08T19:13:36.70+0530 [APP/PROC/WEB/0] OUT at java.lang.Object.wait(Native Method)
2019-05-08T19:13:36.70+0530 [APP/PROC/WEB/0] OUT at java.lang.Object.wait(Object.java:460)
2019-05-08T19:13:36.70+0530 [APP/PROC/WEB/0] OUT at okhttp3.ConnectionPool$1.run(ConnectionPool.java:65)
2019-05-08T19:13:36.70+0530 [APP/PROC/WEB/0] OUT - locked <0x00000000e7f80178> (a okhttp3.ConnectionPool)
2019-05-08T19:13:36.70+0530 [APP/PROC/WEB/0] OUT at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149
I have also tried to reduce the number of threads to 75 but the problem remains. If I can work with 200 threads in my local system (16GB Ram Windows), then why would there be an issue with 75~100 threads in Cloud Foundry?
EDIT
Including the section of the code that creates the threads-
ExecutorService executor = Executors.newFixedThreadPool(100);
for (Object metricObj : metrics) {
JSONObject metric = (JSONObject) metricObj;
String metricTypeId = (String) metric.get("MetricTypeID");
Callable<JSONArray> callable = new metricSearch(metric, type, systemType, odataHandler, systemContextId);
Future<JSONArray> future = executor.submit(callable);
// add Future to the list, we can get return value using Future
list.add(future);
}
executor.shutdown();
So basically, for every object in metrics List (around 1000), I am submitting a job to the Executors, which has a thread limit of 100. So those 1000 jobs are supposed to be executed by a thread pool of 100 threads.
Full thread dump OpenJDK 64-Bit Server VM...). Count the number of threads below that and see how many are being created. It sounds like more are being created than you might think. When you figure out the number it will help to see what limit you might be bumping into. - Daniel MikusametricSearch, is basically a class which creates OkHttpClient clients for each metrics. I have around 1000 metrics. So, is it possible that each new client also internally creates a thread pool of its own, so finally 1000 metrics * 1000 client's thread pool which is causing the issue ? Could it be that ? - Boudhayan Dev