问题
I'm using Spring boot for the application. There is a particular controller when called, makes a couple of independent requests using OkHttpClient. I am using a thread pool of 200 threads to make these calls. It works absolutely fine in the local environment. However, when deployed to cloud foundry, it shows up an error as follows.
I have reduced the number of threads to 100 but the problem still remains. These are the memory settings in my CF instance -
Instance Memory Limit 8192 MB
Memory Quota 3072 MB (available memory 1024 MB)
Disk Quota 1024 MB
This is the error -
2019-05-08T19:13:35.68+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:35.686 INFO 6 --- [ool-2-thread-65] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:35.76+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:35.769 INFO 6 --- [ool-2-thread-62] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:35.81+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:35.813 INFO 6 --- [ool-2-thread-54] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:35.95+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:35.959 INFO 6 --- [ool-2-thread-73] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:35.96+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:35.967 INFO 6 --- [ool-2-thread-60] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:35.96+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:35.967 INFO 6 --- [ool-2-thread-79] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.03+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.034 INFO 6 --- [ool-2-thread-77] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.05+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.054 INFO 6 --- [ool-2-thread-75] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.06+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.066 INFO 6 --- [ool-2-thread-18] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.11+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.119 INFO 6 --- [ool-2-thread-12] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.12+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.121 INFO 6 --- [ool-2-thread-32] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.12+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.123 INFO 6 --- [ool-2-thread-71] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.12+0530 [APP/PROC/WEB/0] ERR Resource exhaustion event: the JVM was unable to create a thread.
2019-05-08T19:13:36.12+0530 [APP/PROC/WEB/0] ERR ResourceExhausted! (1/0)
2019-05-08T19:13:36.13+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.136 INFO 6 --- [ool-2-thread-44] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT | Instance Count | Total Bytes | Class Name |
2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.583 INFO 6 --- [ool-2-thread-92] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.583 INFO 6 --- [ool-2-thread-59] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.583 INFO 6 --- [ool-2-thread-38] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.584 INFO 6 --- [ool-2-thread-25] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT | 100796 | 11914496 | [C |
2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.584 INFO 6 --- [ool-2-thread-78] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.585 INFO 6 --- [ool-2-thread-67] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.585 INFO 6 --- [ol-2-thread-100] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT | 99630 | 2391120 | Ljava/lang/String; |
2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.585 INFO 6 --- [ool-2-thread-80] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.586 INFO 6 --- [ool-2-thread-10] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT | 6940 | 2320424 | [B |
2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT | 41544 | 1661760 | Ljava/util/LinkedHashMap$Entry; |
2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT | 50549 | 1617568 | Ljava/util/HashMap$Node; |
2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 13832 | 1535128 | Ljava/lang/Class; |
2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 42807 | 1369824 | Ljava/util/concurrent/ConcurrentHashMap$Node; |
2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 22775 | 1279152 | [Ljava/util/HashMap$Node; |
2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 21141 | 1183896 | Ljava/util/LinkedHashMap; |
2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 10387 | 914056 | Ljava/lang/reflect/Method; |
2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 11925 | 913696 | [Ljava/lang/Object; |
2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 2202 | 618392 | [I |
2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 36548 | 584768 | Ljava/lang/Object; |
2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 1168 | 485888 | Lcom/sap/LMChatbot/services/metricSearch; |
2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 992 | 372992 | Ljava/lang/Thread; |
2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 200 | 370440 | [Ljava/util/concurrent/ConcurrentHashMap$Node; |
2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 18205 | 291280 | Ljava/util/LinkedHashSet; |
2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 1 | 262160 | [Lorg/h2/mvstore/MVMap; |
2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 8144 | 195456 | Ljava/util/ArrayList; |
2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 2277 | 182544 | [Ljava/lang/ThreadLocal$ThreadLocalMap$Entry; |
2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 7532 | 172640 | [Ljava/lang/Class; |
2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 2964 | 142272 | Lorg/json/simple/JSONObject; |
2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 2177 | 121576 | [Ljava/security/ProtectionDomain; |
2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 2406 | 114776 | [Ljava/lang/String; |
2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 4127 | 99048 | Lorg/springframework/core/MethodClassKey; |
2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 1986 | 95328 | Ljava/util/HashMap; |
2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 845 | 94640 | Ljava/net/SocksSocketImpl; |
2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 2319 | 92760 | Ljava/util/TreeMap$Entry; |
2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 1648 | 92288 | Lokio/Okio$4; |
2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 2185 | 87400 | Ljava/security/AccessControlContext; |
2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 3042 | 73008 | Ljava/util/Collections$UnmodifiableRandomAccessList; |
2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 2242 | 71744 | Ljava/util/LinkedList; |
2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT | 2612 | 62688 | Ljava/util/LinkedList$Node; |
2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT | 826 | 59472 | Lokhttp3/internal/connection/RealConnection; |
2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT | 1846 | 59072 | Ljava/util/concurrent/locks/ReentrantLock$NonfairSync; |
2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT | 801 | 57672 | Lorg/springframework/core/annotation/AnnotationAttributes; |
2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT | 1778 | 56896 | Ljava/net/InetAddress$InetAddressHolder; |
2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.625 INFO 6 --- [ool-2-thread-88] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.625 INFO 6 --- [ool-2-thread-48] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.626 INFO 6 --- [ool-2-thread-64] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.626 INFO 6 --- [ool-2-thread-91] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT | 1417 | 56680 | Ljava/lang/invoke/MethodType; |
2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.626 INFO 6 --- [ool-2-thread-43] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.627 INFO 6 --- [ool-2-thread-86] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.627 INFO 6 --- [ool-2-thread-39] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT | 2277 | 54648 | Ljava/lang/ThreadLocal$ThreadLocalMap; |
2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT | 1671 | 53472 | Ljava/lang/ref/ReferenceQueue; |
2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT | 943 | 52808 | Ljava/lang/Package; |
2019-05-08T19:13:36.63+0530 [APP/PROC/WEB/0] OUT | 1059 | 50832 | Lorg/apache/tomcat/util/modeler/AttributeInfo; |
2019-05-08T19:13:36.63+0530 [APP/PROC/WEB/0] OUT | 884 | 49504 | Lokhttp3/Address; |
2019-05-08T19:13:36.63+0530 [APP/PROC/WEB/0] OUT | 1505 | 48160 | Ljava/util/Hashtable$Entry; |
2019-05-08T19:13:36.63+0530 [APP/PROC/WEB/0] OUT | 998 | 47904 | Ljava/util/concurrent/ThreadPoolExecutor$Worker; |
2019-05-08T19:13:36.63+0530 [APP/PROC/WEB/0] OUT | 984 | 47232 | Lokhttp3/HttpUrl; |
2019-05-08T19:13:36.63+0530 [APP/PROC/WEB/0] OUT | 1419 | 45408 | Ljava/lang/invoke/MethodType$ConcurrentWeakInternSet$WeakEntry; |
2019-05-08T19:13:36.63+0530 [APP/PROC/WEB/0] OUT | 1778 | 42672 | Ljava/net/Inet4Address; |
2019-05-08T19:13:36.63+0530 [APP/PROC/WEB/0] OUT | 1776 | 42624 | Ljava/net/InetSocketAddress$InetSocketAddressHolder; |
2019-05-08T19:13:36.64+0530 [APP/PROC/WEB/0] OUT | 1331 | 42592 | Ljava/util/concurrent/FutureTask; |
2019-05-08T19:13:36.64+0530 [APP/PROC/WEB/0] OUT | 866 | 41568 | Lorg/springframework/core/ResolvableType; |
2019-05-08T19:13:36.64+0530 [APP/PROC/WEB/0] OUT | 10 | 41120 | [Ljava/nio/ByteBuffer; |
2019-05-08T19:13:36.64+0530 [APP/PROC/WEB/0] OUT | 841 | 40368 | Ljava/net/SocketOutputStream; |
2019-05-08T19:13:36.64+0530 [APP/PROC/WEB/0] OUT | 841 | 40368 | Ljava/net/SocketInputStream; |
2019-05-08T19:13:36.64+0530 [APP/PROC/WEB/0] OUT | 997 | 39880 | Ljava/lang/ref/Finalizer; |
2019-05-08T19:13:36.64+0530 [APP/PROC/WEB/0] OUT | 1649 | 39576 | Lokio/Buffer; |
2019-05-08T19:13:36.64+0530 [APP/PROC/WEB/0] OUT | 819 | 39312 | Lch/qos/logback/classic/Logger; |
2019-05-08T19:13:36.64+0530 [APP/PROC/WEB/0] OUT | 960 | 38400 | Lorg/springframework/util/ConcurrentReferenceHashMap$Segment; |
2019-05-08T19:13:36.65+0530 [APP/PROC/WEB/0] OUT | 960 | 36544 | [Lorg/springframework/util/ConcurrentReferenceHashMap$Reference; |
2019-05-08T19:13:36.65+0530 [APP/PROC/WEB/0] OUT | 401 | 36336 | [Ljava/util/Hashtable$Entry; |
2019-05-08T19:13:36.65+0530 [APP/PROC/WEB/0] OUT | 1135 | 36320 | Lsun/security/util/DerInputBuffer; |
2019-05-08T19:13:36.65+0530 [APP/PROC/WEB/0] OUT | 1135 | 36320 | Lsun/security/util/DerValue; |
2019-05-08T19:13:36.65+0530 [APP/PROC/WEB/0] OUT | 560 | 35840 | Ljava/util/concurrent/ConcurrentHashMap; |
2019-05-08T19:13:36.65+0530 [APP/PROC/WEB/0] OUT | 743 | 35664 | Lorg/springframework/util/ConcurrentReferenceHashMap$SoftEntryReference; |
2019-05-08T19:13:36.65+0530 [APP/PROC/WEB/0] OUT | 887 | 35480 | Ljava/math/BigInteger; |
2019-05-08T19:13:36.65+0530 [APP/PROC/WEB/0] OUT | 886 | 35440 | Lokhttp3/ConnectionPool; |
2019-05-08T19:13:36.65+0530 [APP/PROC/WEB/0] OUT | 1413 | 33912 | Ljava/util/jar/Attributes$Name; |
2019-05-08T19:13:36.66+0530 [APP/PROC/WEB/0] OUT | 134 | 33712 | [[C |
2019-05-08T19:13:36.66+0530 [APP/PROC/WEB/0] OUT | 1351 | 32424 | Lorg/json/simple/JSONArray; |
2019-05-08T19:13:36.66+0530 [APP/PROC/WEB/0] OUT | 184 | 32384 | Lorg/springframework/context/annotation/ConfigurationClassBeanDefinitionReader$ConfigurationClassBeanDefinition; |
2019-05-08T19:13:36.66+0530 [APP/PROC/WEB/0] OUT | 1295 | 31080 | Ljava/util/ArrayDeque; |
2019-05-08T19:13:36.66+0530 [APP/PROC/WEB/0] OUT | 904 | 28928 | Ljava/io/FileDescriptor; |
2019-05-08T19:13:36.66+0530 [APP/PROC/WEB/0] OUT | 1776 | 28416 | Ljava/net/InetSocketAddress; |
2019-05-08T19:13:36.66+0530 [APP/PROC/WEB/0] OUT | 580 | 27840 | Ljava/util/concurrent/locks/ReentrantReadWriteLock$NonfairSync; |
2019-05-08T19:13:36.66+0530 [APP/PROC/WEB/0] OUT | 312 | 27456 | Lorg/h2/table/Column; |
2019-05-08T19:13:36.66+0530 [APP/PROC/WEB/0] OUT | 1135 | 27240 | Lsun/security/util/DerInputStream; |
2019-05-08T19:13:36.67+0530 [APP/PROC/WEB/0] OUT | 841 | 26912 | Ljava/net/Socket; |
2019-05-08T19:13:36.67+0530 [APP/PROC/WEB/0] OUT | 1675 | 26800 | Ljava/lang/ref/ReferenceQueue$Lock; |
2019-05-08T19:13:36.67+0530 [APP/PROC/WEB/0] OUT | 1114 | 26736 | Lsun/security/x509/AVA; |
2019-05-08T19:13:36.67+0530 [APP/PROC/WEB/0] OUT | 1114 | 26736 | [Lsun/security/x509/AVA; |
2019-05-08T19:13:36.67+0530 [APP/PROC/WEB/0] OUT | 1114 | 26736 | Lsun/security/x509/RDN; |
2019-05-08T19:13:36.67+0530 [APP/PROC/WEB/0] OUT | 1075 | 25800 | Ljava/util/concurrent/LinkedBlockingQueue$Node; |
2019-05-08T19:13:36.67+0530 [APP/PROC/WEB/0] OUT | 447 | 25032 | Ljava/lang/invoke/MemberName; |
2019-05-08T19:13:36.67+0530 [APP/PROC/WEB/0] OUT | 598 | 23920 | Lokhttp3/internal/http/RealInterceptorChain; |
2019-05-08T19:13:36.67+0530 [APP/PROC/WEB/0] OUT | 297 | 23760 | Ljava/lang/reflect/Constructor; |
2019-05-08T19:13:36.68+0530 [APP/PROC/WEB/0] OUT | 140 | 23520 | Lorg/springframework/beans/factory/support/RootBeanDefinition; |
2019-05-08T19:13:36.68+0530 [APP/PROC/WEB/0] OUT | 1443 | 23088 | Ljava/util/LinkedHashMap$LinkedKeySet; |
2019-05-08T19:13:36.68+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.682 INFO 6 --- [ool-2-thread-83] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.68+0530 [APP/PROC/WEB/0] OUT | 960 | 23040 | Lorg/springframework/util/ConcurrentReferenceHashMap$ReferenceManager; |
2019-05-08T19:13:36.68+0530 [APP/PROC/WEB/0] OUT | 477 | 22896 | Lorg/apache/tomcat/util/modeler/OperationInfo; |
2019-05-08T19:13:36.68+0530 [APP/PROC/WEB/0] OUT | 290 | 22744 | [Z |
2019-05-08T19:13:36.68+0530 [APP/PROC/WEB/0] OUT | 566 | 22640 | Ljava/lang/ref/SoftReference; |
2019-05-08T19:13:36.68+0530 [APP/PROC/WEB/0] OUT | 1332 | 21312 | Lorg/springframework/core/annotation/AnnotationUtils$DefaultValueHolder; |
2019-05-08T19:13:36.68+0530 [APP/PROC/WEB/0] OUT | 884 | 21216 | Lokhttp3/Route; |
2019-05-08T19:13:36.69+0530 [APP/PROC/WEB/0] OUT | 882 | 21168 | Ljava/net/Proxy; |
2019-05-08T19:13:36.69+0530 [APP/PROC/WEB/0] OUT | 326 | 20864 | Ljava/net/URL; |
2019-05-08T19:13:36.69+0530 [APP/PROC/WEB/0] OUT | 630 | 20160 | Ljavax/management/MBeanAttributeInfo; |
2019-05-08T19:13:36.69+0530 [APP/PROC/WEB/0] OUT | 824 | 19776 | Lokio/RealBufferedSource; |
2019-05-08T19:13:36.69+0530 [APP/PROC/WEB/0] OUT | 824 | 19776 | Lokio/Okio$2; |
2019-05-08T19:13:36.69+0530 [APP/PROC/WEB/0] OUT | 824 | 19776 | Lokio/Okio$1; |
2019-05-08T19:13:36.69+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.695 INFO 6 --- [ool-2-thread-56] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
2019-05-08T19:13:36.69+0530 [APP/PROC/WEB/0] OUT | 824 | 19776 | Lokio/RealBufferedSink; |
2019-05-08T19:13:36.69+0530 [APP/PROC/WEB/0] ERR ERROR: PoolStats action failed: cannot determine memory usage statistics since the JVM is unable to create a thread
2019-05-08T19:13:36.69+0530 [APP/PROC/WEB/0] ERR jvmkill killing current process
2019-05-08T19:13:36.70+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36
2019-05-08T19:13:36.70+0530 [APP/PROC/WEB/0] OUT Full thread dump OpenJDK 64-Bit Server VM (25.212-b03 mixed mode):
2019-05-08T19:13:36.70+0530 [APP/PROC/WEB/0] OUT "OkHttp ConnectionPool" #2406 daemon prio=5 os_prio=0 tid=0x00005641e28d4000 nid=0x467 in Object.wait() [0x00007ff04cd4f000]
2019-05-08T19:13:36.70+0530 [APP/PROC/WEB/0] OUT java.lang.Thread.State: TIMED_WAITING (on object monitor)
2019-05-08T19:13:36.70+0530 [APP/PROC/WEB/0] OUT at java.lang.Object.wait(Native Method)
2019-05-08T19:13:36.70+0530 [APP/PROC/WEB/0] OUT at java.lang.Object.wait(Object.java:460)
2019-05-08T19:13:36.70+0530 [APP/PROC/WEB/0] OUT at okhttp3.ConnectionPool$1.run(ConnectionPool.java:65)
2019-05-08T19:13:36.70+0530 [APP/PROC/WEB/0] OUT - locked <0x00000000e7f80178> (a okhttp3.ConnectionPool)
2019-05-08T19:13:36.70+0530 [APP/PROC/WEB/0] OUT at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149
I have also tried to reduce the number of threads to 75 but the problem remains. If I can work with 200 threads in my local system (16GB Ram Windows), then why would there be an issue with 75~100 threads in Cloud Foundry?
EDIT
Including the section of the code that creates the threads-
ExecutorService executor = Executors.newFixedThreadPool(100);
for (Object metricObj : metrics) {
JSONObject metric = (JSONObject) metricObj;
String metricTypeId = (String) metric.get("MetricTypeID");
Callable<JSONArray> callable = new metricSearch(metric, type, systemType, odataHandler, systemContextId);
Future<JSONArray> future = executor.submit(callable);
// add Future to the list, we can get return value using Future
list.add(future);
}
executor.shutdown();
So basically, for every object in metrics List (around 1000), I am submitting a job to the Executors, which has a thread limit of 100. So those 1000 jobs are supposed to be executed by a thread pool of 100 threads.
回答1:
So, each of these threads make a REST API call. I was creating a new instance of OkHttpClient for each thread. As it turns out, each client instance would spawn many threads of its own internally. So the total number of threads were more than what I specified.
After sharing the same instance for the OkHttpClient client with all the threads, the issue seems to be resolved.
来源：https://stackoverflow.com/questions/56042466/jvm-unable-to-create-a-thread-in-cloud-foundry