High CPU utilization for threads that seem to be waiting

Question

I\'m currently running some JMeter tests to test the performance of a web service. It\'s using a very high amount of CPU. For one JMeter request thread, it\'s using anywhere

Answer 1

As Brendan Gregg points out the blog article you linked, hprof samples from all threads the JVM considers runnable. As you can see in the Javadoc of Thread.state, the JVM distinguishes the following thread states:

• NEW: A thread that has not yet started is in this state.
• RUNNABLE: A thread executing in the Java virtual machine is in this state.
• BLOCKED: A thread that is blocked waiting for a monitor lock is in this state.
• WAITING: A thread that is waiting indefinitely for another thread to perform a particular action is in this state.
• TIMED_WAITING: A thread that is waiting for another thread to perform an action for up to a specified waiting time is in this state.
• TERMINATED: A thread that has exited is in this state.

As we can see, the JVM does not have a dedicated state for a thread waiting for I/O. That's because such a thread is actually blocked by the operating system, not the JVM. That is, as far the JVM is concerned, a thread waiting for the network adapter is runnable. Indeed, the detail Javadoc for the RUNNABLE state writes:

Thread state for a runnable thread. A thread in the runnable state is executing in the Java virtual machine but it may be waiting for other resources from the operating system such as processor.

Therefore, the presence of I/O methods in a hprof "cpu" sampling does not imply that these methods consumed CPU, as their wait time is counted as well.

You can either:

• assume that I/O methods are not responsible for the CPU consumption, and focus on the other methods
• use a better profiler that takes into account waiting for OS level resources