Jmap can't connect to make a dump

Question

We have an open beta of an app which occasionally causes the heapspace to overflow. The JVM reacts by going on a permanent vacation.

To analyze this I would like to

Answer 1

In my case it is not as simple as check the user :(

I have a script called collectd-java which invokes jstat and jmap. I've checked by top that such script is launched, as expected, by the user owning the JVM. However, jstat gives me what I need and jmap can't attach. Here is the script - the echo stuff is just the format I need to present the values:

HOSTNAME="${COLLECTD_HOSTNAME:-localhost}"
INTERVAL="${COLLECTD_INTERVAL:-60}"
MAIN_CLASS="my.fully.qualified.MainClass"
PID=$(pgrep -f ${MAIN_CLASS})

get_jstat_classloaderdata() {
VALUE=`jstat -class $PID 1 1 | awk '{print $1}' | grep -vi loaded`
echo "PUTVAL \"$HOSTNAME/exec-cecoco/gauge-java_classloader_loaded\" interval=$INTERVAL N:$VALUE"

VALUE=`jstat -class $PID 1 1 | awk '{print $2}' | grep -vi bytes`
echo "PUTVAL \"$HOSTNAME/exec-cecoco/gauge-java_classloader_bytesload\" interval=$INTERVAL N:$VALUE"

VALUE=`jstat -class $PID 1 1 | awk '{print $3}' | grep -vi unload`
echo "PUTVAL \"$HOSTNAME/exec-cecoco/gauge-java_classloader_unloaded\" interval=$INTERVAL N:$VALUE"

VALUE=`jstat -class $PID 1 1 | awk '{print $4}' | grep -vi bytes`
echo "PUTVAL \"$HOSTNAME/exec-cecoco/gauge-java_classloader_bytesunload\" interval=$INTERVAL N:$VALUE"

VALUE=`jstat -class $PID 1 1 | awk '{print $5}' | grep -vi time`
echo "PUTVAL \"$HOSTNAME/exec-cecoco/gauge-java_classloader_time\" interval=$INTERVAL N:$VALUE"
}

get_jmap_heapdata() {
        VALUE=$(jmap -heap ${PID} | grep MinHeapFreeRatio |awk '{print $3}')
        echo "PUTVAL \"$HOSTNAME/exec-cecoco/gauge-jmap_minheapfreeratio\" interval=$INTERVAL N:$VALUE"

        VALUE=$(jmap -heap ${PID} | grep   MaxHeapFreeRatio|awk '{print $3}')
        echo "PUTVAL \"$HOSTNAME/exec-cecoco/gauge-jmap_maxheapfreeratio\" interval=$INTERVAL N:$VALUE"

        VALUE=$(jmap -heap ${PID} | grep   MaxHeapSize|awk '{print $3}')
        echo "PUTVAL \"$HOSTNAME/exec-cecoco/gauge-jmap_maxheapsize\" interval=$INTERVAL N:$VALUE"
}
##Do it
get_jmap_heapdata
get_jstat_classloaderdata

Jstat succeeds and jmap fails. Does anyone understands it ?

Answer 2

If someone tries to get Heap Dump of Java application in Docker container. This is the only solution that worked for me:

docker exec <container-name> jcmd 1 GC.heap_dump /tmp/docker.hprof

It basically dumps the heap of process with pid=1 using jcmd

See https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr006.html

Answer 3

1.Execute "Docker ps", will give the container Id of all services and collect the container id foe TSC.
2.Execute "docker exec -it CONTAINER_ID bash" (replace CONTAINER_ID with TSC Container id)
3.Bash will come and then execute the "jps" on bash, that will give you the PID for process(it will be 1 for jar)
4.Execute the "jstack PID > threadDump.tdump"(replace PID with process id received in step 3, it should be 1)
5.Execute the "jmap -dump:format=b,file=heapDump.hprof PID"(replace PID with process id received in step 3, it should be 1)
6.Then we have to exit the bash using "exit" command
7.Execute "sudo docker cp CONTAINER_ID:heapDump.hprof ." from ec2 command line, that will copy the dump file on ec2 machine present working directory.
8.Execute "sudo docker cp CONTAINER_ID:threadDump.tdump ." from ec2 command line, that will copy the dump file on ec2 machine present working directory.

Answer 4

I was running the jmap and the application with the same user and still get the error.

The solution was run that comand before the jmap

echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope

Than is just use jmap and will works fine

jmap -heap 17210

Answer 5

What worked for me was to simply issue the command with sudo as in:

sudo jmap -heap 21797

Answer 6

Not sure why a plain "jmap " fails when I docker exec -it into my container running centos7 systemd and a java service, but below jmap options worked for me. Thanks: https://dkbalachandar.wordpress.com/2016/07/05/thread-dump-from-a-docker-container/

[root@b29924306cfe /]# jmap 170 Attaching to process ID 170, please wait... Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process: ptrace(PTRACE_ATTACH, ..) failed for 170: Operation not permitted sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process: ptrace(PTRACE_ATTACH, ..) failed for 170: Operation not permitted

[root@b29924306cfe /]# jmap -dump:live,format=b,file=heapDump.hprof 170 Dumping heap to /heapDump.hprof ... Heap dump file created