java run out of memory issue

问题:

I have a jar file running on an amazon-ec2-m1.large instance with linux 64bit operating system. I run out of memory after different hours, usually between 2-4, although in my last log write (before the error file created ) I see the following:

Memory -       totalHeapSize: 264 MB,       maxHeapSize: 1656 MB,       freeHeapSize: 220 MB 

When I look from the jar start until in run out of memory - all values are the same and the freeHeapSize is between 200-230 MB (i check every 30 seconds and do System.gc() )

in the hs_err_pid2250.log wrriten:

    #     # There is insufficient memory for the Java Runtime Environment to continue.     # Cannot create GC thread. Out of system resources.     # Possible reasons:     #   The system is out of physical RAM or swap space     #   In 32 bit mode, the process size limit was hit     # Possible solutions:     #   Reduce memory load on the system     #   Increase physical memory or swap space     #   Check if swap backing store is full     #   Use 64 bit Java on a 64 bit OS     #   Decrease Java heap size (-Xmx/-Xms)     #   Decrease number of Java threads     #   Decrease Java thread stack sizes (-Xss)     #   Set larger code cache with -XX:ReservedCodeCacheSize=     # This output file may be truncated or incomplete.     #     #  Out of Memory Error (gcTaskThread.cpp:46), pid=2250, tid=140227064182528     #     # JRE version: 7.0_25-b15     # Java VM: Java HotSpot(TM) 64-Bit Server VM (23.25-b01 mixed mode linux-amd64 compressed oops)     # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again     #      ---------------  T H R E A D  ---------------      Current thread (0x00007f8920007800):  JavaThread "Unknown thread" [_thread_in_vm, id=2251, stack(0x00007f8928493000,0x00007f8928594000)]      Stack: [0x00007f8928493000,0x00007f8928594000],  sp=0x00007f89285925d0,  free space=1021k     Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)     V  [libjvm.so+0x8a1d3a]  VMError::report_and_die()+0x2ea     V  [libjvm.so+0x4071fb]  report_vm_out_of_memory(char const*, int, unsigned long, char const*)+0x9b     V  [libjvm.so+0x4b3e2a]  GCTaskThread::GCTaskThread(GCTaskManager*, unsigned int, unsigned int)+0xfa     V  [libjvm.so+0x4b2896]  GCTaskManager::initialize()+0x266     V  [libjvm.so+0x75dc4f]  ParallelScavengeHeap::initialize()+0x6cf     V  [libjvm.so+0x87a078]  Universe::initialize_heap()+0xb8     V  [libjvm.so+0x87c7ed]  universe_init()+0x7d     V  [libjvm.so+0x500315]  init_globals()+0x65     V  [libjvm.so+0x863a62]  Threads::create_vm(JavaVMInitArgs*, bool*)+0x1e2     V  [libjvm.so+0x55ff34]  JNI_CreateJavaVM+0x74     C  [libjli.so+0x2f8e]  JavaMain+0x9e       ---------------  P R O C E S S  ---------------      Java Threads: ( => current thread )      Other Threads:      =>0x00007f8920007800 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=2251, stack(0x00007f8928493000,0x00007f8928594000)]      VM state:not at safepoint (not fully initialized)      VM Mutex/Monitor currently owned by a thread: None      GC Heap History (0 events):     No events      Deoptimization events (0 events):     No events      Internal exceptions (0 events):     No events      Events (0 events):     No events       Dynamic libraries:     00400000-00401000 r-xp 00000000 ca:01 9384                               /usr/java/jdk1.7.0_25/bin/java     00600000-00601000 rw-p 00000000 ca:01 9384                               /usr/java/jdk1.7.0_25/bin/java     01f54000-01f75000 rw-p 00000000 00:00 0                                  [heap]     86600000-87ac0000 rw-p 00000000 00:00 0      87ac0000-8b800000 rw-p 00000000 00:00 0      8b800000-905b0000 rw-p 00000000 00:00 0      905b0000-d92b0000 rw-p 00000000 00:00 0      d92b0000-db980000 rw-p 00000000 00:00 0      db980000-100000000 rw-p 00000000 00:00 0      7f891d000000-7f891d270000 rwxp 00000000 00:00 0      7f891d270000-7f8920022000 rw-p 00000000 00:00 0      7f8920022000-7f8924000000 ---p 00000000 00:00 0      7f8925555000-7f892574d000 rw-p 00000000 00:00 0      7f892574d000-7f892574e000 ---p 00000000 00:00 0      7f892574e000-7f892584e000 rw-p 00000000 00:00 0      7f892584e000-7f892584f000 ---p 00000000 00:00 0      7f892584f000-7f892595a000 rw-p 00000000 00:00 0                          [stack:2252]     7f892595a000-7f8925978000 rw-p 00000000 00:00 0      7f8925978000-7f892599f000 rw-p 00000000 00:00 0      7f892599f000-7f8925be6000 rw-p 00000000 00:00 0      7f8925be6000-7f8925bf1000 rw-p 00000000 00:00 0      7f8925bf1000-7f8925c0f000 rw-p 00000000 00:00 0      7f8925c0f000-7f8925c36000 rw-p 00000000 00:00 0      7f8925c36000-7f8925e7c000 rw-p 00000000 00:00 0      7f8925e7c000-7f8925e90000 rw-p 00000000 00:00 0      7f8925e90000-7f8925fb3000 rw-p 00000000 00:00 0      7f8925fb3000-7f8925fb4000 rw-p 00000000 00:00 0      7f8925fb4000-7f8925fcf000 r-xp 00000000 ca:01 17790                      /usr/java/jdk1.7.0_25/jre/lib/amd64/libzip.so     7f8925fcf000-7f89261ce000 ---p 0001b000 ca:01 17790                      /usr/java/jdk1.7.0_25/jre/lib/amd64/libzip.so     7f89261ce000-7f89261cf000 rw-p 0001a000 ca:01 17790                      /usr/java/jdk1.7.0_25/jre/lib/amd64/libzip.so     7f89261cf000-7f89261db000 r-xp 00000000 ca:01 17544                      /lib64/libnss_files-2.12.so     7f89261db000-7f89263da000 ---p 0000c000 ca:01 17544                      /lib64/libnss_files-2.12.so     7f89263da000-7f89263db000 r--p 0000b000 ca:01 17544                      /lib64/libnss_files-2.12.so     7f89263db000-7f89263dc000 rw-p 0000c000 ca:01 17544                      /lib64/libnss_files-2.12.so     7f89263dc000-7f8926405000 r-xp 00000000 ca:01 17571                      /usr/java/jdk1.7.0_25/jre/lib/amd64/libjava.so     7f8926405000-7f8926605000 ---p 00029000 ca:01 17571                      /usr/java/jdk1.7.0_25/jre/lib/amd64/libjava.so     7f8926605000-7f8926607000 rw-p 00029000 ca:01 17571                      /usr/java/jdk1.7.0_25/jre/lib/amd64/libjava.so     7f8926607000-7f8926614000 r-xp 00000000 ca:01 17789                      /usr/java/jdk1.7.0_25/jre/lib/amd64/libverify.so     7f8926614000-7f8926813000 ---p 0000d000 ca:01 17789                      /usr/java/jdk1.7.0_25/jre/lib/amd64/libverify.so     7f8926813000-7f8926815000 rw-p 0000c000 ca:01 17789                      /usr/java/jdk1.7.0_25/jre/lib/amd64/libverify.so     7f8926815000-7f892681c000 r-xp 00000000 ca:01 17559                      /lib64/librt-2.12.so     7f892681c000-7f8926a1b000 ---p 00007000 ca:01 17559                      /lib64/librt-2.12.so     7f8926a1b000-7f8926a1c000 r--p 00006000 ca:01 17559                      /lib64/librt-2.12.so     7f8926a1c000-7f8926a1d000 rw-p 00007000 ca:01 17559                      /lib64/librt-2.12.so     7f8926a1d000-7f8926a9f000 r-xp 00000000 ca:01 17537                      /lib64/libm-2.12.so     7f8926a9f000-7f8926c9e000 ---p 00082000 ca:01 17537                      /lib64/libm-2.12.so     7f8926c9e000-7f8926c9f000 r--p 00081000 ca:01 17537                      /lib64/libm-2.12.so     7f8926c9f000-7f8926ca0000 rw-p 00082000 ca:01 17537                      /lib64/libm-2.12.so     7f8926ca0000-7f89276e1000 r-xp 00000000 ca:01 17795                      /usr/java/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so     7f89276e1000-7f89278e1000 ---p 00a41000 ca:01 17795                      /usr/java/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so     7f89278e1000-7f8927984000 rw-p 00a41000 ca:01 17795                      /usr/java/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so     7f8927984000-7f89279c0000 rw-p 00000000 00:00 0      7f89279c0000-7f8927b41000 r-xp 00000000 ca:01 7616                       /lib64/libc-2.12.so     7f8927b41000-7f8927d41000 ---p 00181000 ca:01 7616                       /lib64/libc-2.12.so     7f8927d41000-7f8927d45000 r--p 00181000 ca:01 7616                       /lib64/libc-2.12.so     7f8927d45000-7f8927d46000 rw-p 00185000 ca:01 7616                       /lib64/libc-2.12.so     7f8927d46000-7f8927d4c000 rw-p 00000000 00:00 0      7f8927d4c000-7f8927d4e000 r-xp 00000000 ca:01 17535                      /lib64/libdl-2.12.so     7f8927d4e000-7f8927f4e000 ---p 00002000 ca:01 17535                      /lib64/libdl-2.12.so     7f8927f4e000-7f8927f4f000 r--p 00002000 ca:01 17535                      /lib64/libdl-2.12.so     7f8927f4f000-7f8927f50000 rw-p 00003000 ca:01 17535                      /lib64/libdl-2.12.so     7f8927f50000-7f8927f66000 r-xp 00000000 ca:01 16704                      /usr/java/jdk1.7.0_25/jre/lib/amd64/jli/libjli.so     7f8927f66000-7f8928166000 ---p 00016000 ca:01 16704                      /usr/java/jdk1.7.0_25/jre/lib/amd64/jli/libjli.so     7f8928166000-7f8928167000 rw-p 00016000 ca:01 16704                      /usr/java/jdk1.7.0_25/jre/lib/amd64/jli/libjli.so     7f8928167000-7f892817e000 r-xp 00000000 ca:01 7640                       /lib64/libpthread-2.12.so     7f892817e000-7f892837d000 ---p 00017000 ca:01 7640                       /lib64/libpthread-2.12.so     7f892837d000-7f892837e000 r--p 00016000 ca:01 7640                       /lib64/libpthread-2.12.so     7f892837e000-7f892837f000 rw-p 00017000 ca:01 7640                       /lib64/libpthread-2.12.so     7f892837f000-7f8928383000 rw-p 00000000 00:00 0      7f8928383000-7f89283a1000 r-xp 00000000 ca:01 9381                       /lib64/ld-2.12.so     7f89283cb000-7f89283d5000 rw-p 00000000 00:00 0      7f89283d5000-7f892848b000 rw-p 00000000 00:00 0      7f892848b000-7f8928493000 rw-s 00000000 ca:01 8174                       /tmp/hsperfdata_gurustrade/2250     7f8928493000-7f8928496000 ---p 00000000 00:00 0      7f8928496000-7f8928598000 rw-p 00000000 00:00 0                          [stack:2251]     7f892859b000-7f892859e000 rw-p 00000000 00:00 0      7f892859e000-7f892859f000 r--p 00000000 00:00 0      7f892859f000-7f89285a0000 rw-p 00000000 00:00 0      7f89285a0000-7f89285a1000 r--p 0001d000 ca:01 9381                       /lib64/ld-2.12.so     7f89285a1000-7f89285a2000 rw-p 0001e000 ca:01 9381                       /lib64/ld-2.12.so     7f89285a2000-7f89285a3000 rw-p 00000000 00:00 0      7fff64527000-7fff64548000 rw-p 00000000 00:00 0                          [stack]     7fff645ff000-7fff64600000 r-xp 00000000 00:00 0                          [vdso]     ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]      VM Arguments:     java_command: /home/gurustrade/proj/versions/trunk/gt_engine/gt_engine.jar     Launcher Type: SUN_STANDARD      Environment Variables:     PATH=/usr/bin:/bin     SHELL=/bin/sh      Signal Handlers:     SIGSEGV: [libjvm.so+0x8a26a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004     SIGBUS: [libjvm.so+0x8a26a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004     SIGFPE: [libjvm.so+0x73f3d0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004     SIGPIPE: [libjvm.so+0x73f3d0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004     SIGXFSZ: [libjvm.so+0x73f3d0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004     SIGILL: [libjvm.so+0x73f3d0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004     SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000     SIGUSR2: [libjvm.so+0x73ed20], sa_mask[0]=0x00000000, sa_flags=0x10000004     SIGHUP: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000     SIGINT: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000     SIGTERM: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000     SIGQUIT: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000       ---------------  S Y S T E M  ---------------      OS:Linux     uname:Linux 3.4.48-45.46.amzn1.x86_64 #1 SMP Wed Jun 12 02:04:44 UTC 2013 x86_64     libc:glibc 2.12 NPTL 2.12      rlimit: STACK 8192k, CORE 0k, NPROC 1024, NOFILE 4096, AS infinity     load average:1.01 1.04 1.10      /proc/meminfo:     MemTotal:        7633612 kB     MemFree:         5784036 kB     Buffers:          186424 kB     Cached:          1025752 kB     SwapCached:            0 kB     Active:           980284 kB     Inactive:         706036 kB     Active(anon):     474096 kB     Inactive(anon):      100 kB     Active(file):     506188 kB     Inactive(file):   705936 kB     Unevictable:           0 kB     Mlocked:               0 kB     SwapTotal:             0 kB     SwapFree:              0 kB     Dirty:                88 kB     Writeback:             0 kB     AnonPages:        474084 kB     Mapped:            43752 kB     Shmem:               156 kB     Slab:             103892 kB     SReclaimable:      84548 kB     SUnreclaim:        19344 kB     KernelStack:        9032 kB     PageTables:        20180 kB     NFS_Unstable:          0 kB     Bounce:                0 kB     WritebackTmp:          0 kB     CommitLimit:     3816804 kB     Committed_AS:    2130476 kB     VmallocTotal:   34359738367 kB     VmallocUsed:       23908 kB     VmallocChunk:   34359714455 kB     AnonHugePages:         0 kB     HugePages_Total:       0     HugePages_Free:        0     HugePages_Rsvd:        0     HugePages_Surp:        0     Hugepagesize:       2048 kB     DirectMap4k:     7872512 kB     DirectMap2M:           0 kB       CPU:total 2 (4 cores per cpu, 1 threads per core) family 6 model 23 stepping 10, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, tsc      /proc/cpuinfo:     processor   : 0     vendor_id   : GenuineIntel     cpu family  : 6     model       : 23     model name  : Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz     stepping    : 10     microcode   : 0xa07     cpu MHz     : 2660.000     cache size  : 6144 KB     fpu     : yes     fpu_exception   : yes     cpuid level : 13     wp      : yes     flags       : fpu tsc msr pae cx8 cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm dtherm tpr_shadow vnmi flexpriority     bogomips    : 5320.00     clflush size    : 64     cache_alignment : 64     address sizes   : 38 bits physical, 48 bits virtual     power management:      processor   : 1     vendor_id   : GenuineIntel     cpu family  : 6     model       : 23     model name  : Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz     stepping    : 10     microcode   : 0xa07     cpu MHz     : 2660.000     cache size  : 6144 KB     fpu     : yes     fpu_exception   : yes     cpuid level : 13     wp      : yes     flags       : fpu tsc msr pae cx8 cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm dtherm tpr_shadow vnmi flexpriority     bogomips    : 5320.00     clflush size    : 64     cache_alignment : 64     address sizes   : 38 bits physical, 48 bits virtual     power management:        Memory: 4k page, physical 7633612k(5784036k free), swap 0k(0k free)      vm_info: Java HotSpot(TM) 64-Bit Server VM (23.25-b01) for linux-amd64 JRE (1.7.0_25-b15),   built on Jun  5 2013 20:48:36 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)      time: Mon Aug  5 16:11:16 2013     elapsed time: 0 seconds 

Also, I use threadpoolexecutor with 30 core threads and 100 max (never user more than 30 although at some times I use all 30)

Finally, I user bonecp connection pool with 50 connections

Any suggestions? :)

After suggestion, I added a heap dump after running the application from my local pc. How can I continue from here? Are the critical orange jumps important?, or just the used important?

回答1:

I was able to resolve an almost identical error by following the advice given by:

https://confluence.csiro.au/pages/viewpage.action?pageId=278167841

Essentially, they state:

When the Java Virtual Machine starts, by default it spawns a number of Garbage Collection (GC) threads, which are used for parallel GC operations … the number of such threads is calculated by this formula: (ncpus <= 8) ? ncpus : 3 + ((ncpus * 5) / 8) … Because of the creation of these many threads … the program will automatically reach the system limit …

I limited the number of garbage collecting threads using the environment variable, like so:

export _JAVA_OPTIONS="-XX:ParallelGCThreads=2" 

Not sure about the overall performance hit regarding GC, but at least today's work is done.

Please check the original post for more options.

Good Luck, - Stu

回答2:

Please try 64 bit version of Java if you are using 32 bit version,

Also add new JVM argument , "-XX:+CMSClassUnloadingEnabled \ " which clear some unused class objects

回答3:

If you have more physical ram available use that . It seems that the heap size allocated is only 1656 mb which perhaps is not enough. Try running java jar file with two switches -xmx4096mb and -xms 2048 Then monitor the usage , perhaps the memory requirement is too large for your application , if you still run out of memory after sometime then it needs further investigation to check if your code is leaking memory. Hope this helps , let me know if you need more clarification

回答4:

Heap is only part of the equation. Have a look at total resident memory - it includes heap and off heap contributions. Off heap includes mapped JARs, thread stacks (~1MB per thread), perm gen, etc. SO has a number of questions on how to do it on Linux.

回答5:

Without trying to decode the dump data, I'll observe that there are basically three cases for running a JVM out of memory:

1. Having a structure (list of lists or whatnot) that is allowed to accumulate masses of unused (but still referenced) data.
2. Leaving various "handles" open -- file handles, threads, data base operations. Especially if using an API with a native component you can potentially leave a lot of stuff outside of the reach of GC.
3. Making a single extremely large request (or a "burst" of such requests).

Kind of in-between here would be classes. If you run a app that dynamically creates and loads classes, you must properly use individual class loaders to make the classes collectable.

文章来源: java run out of memory issue