jmx/jstatd access to remote machine through an ssh tunnel

Question

I'd like to use visualvm app through an ssh tunnel (to an EC2 machine) using jmx OR jstatd. How do I do this? Here is a list of what has been tried (and failed): (BTW: if visual vm is not appropriate, how do I find memory leaks on a remote machine?)

jstatd: attempt:

I set up the jstatd server on the EC2 machine (app was already running on it) Then I set up a tunnel mapping local port 3333 to remote port 1099 In VisualVM I tried to connect using jstatd on port 3333 ... none of the processes on EC2 showed up

This link says that jstatd opens up another port: http://rukuro-blog.heroku.com/2011/06/30/monitoring-remote-java-applications-with-visualvm ... so I tunneled that port from local host to remote. I restarted visual vm ... still nothing

When I tried listing processes running from EC2 using the folloing command:

jps -l -m -v rmi://localhost

... I got a list of processes

When I listed it on my home machine using

jps -l -m -v rmi://localhost:3333

.... I got none ! So, is the rmi port not being tunnelled with jstatd port (3333) ... ?

the jmx attempt:

I started the application on the remote machine with the following command:

java -Dcom.sun.management.jmxremote.port=3333 \
     -Dcom.sun.management.jmxremote.ssl=false \
     -Dcom.sun.management.jmxremote.authenticate=false\
        -cp :post/* <appName>

... the app works so I tunnelled local port 3333 to remote 3333

Then I tried setting up a jmx link to localhost:3333 from visual vm interface: ... it pops up an error saying:

cannot connect to localhost:3333 using service:jmx:rmi:///jndi/rmi://localhost:3333/jmxrmi

If I need to set up a link to the remove RMI server I'm not sure how to do it.

JMXMP attempt:

This holds promise but something is not quite right:

First I added the jmx_remote jar to the classpath and started the application on the remote machine using the same command as in the JMX case I showed above. I found the jar at oracle download link

I setup up an ssh tunnel from local port 3333 to remote port 3333. Then I started up visualvm with the same jmx_remote file in the calsspath.

visualvm -cp:a ~/jmx/jmxremote_optional.jar

Then I tried to connect visualvm to the remote server with:

service:jmx:jmxmp://localhost:3333

Now visual vm seems to try to connect indefinitely. It shows "Adding service:jmx:jmxp://localhost:3333" in the status bar ... and continutes to do so till I shutdown the application on the remote end at which time it throws a popup saying that it couldnt' connect with the server.

On using SOCKS:

My attempts at using both JMX and jstatd failed.

I'm not sure the SOCKS proxy is working, so here is how I tried to connect:

ssh -i ~/.ssh/starter.pem -v -D 9696 user@host

Just for completeness, I started the command on the other end with:

    java -Dcom.sun.management.jmxremote\
 -Dcom.sun.management.jmxremote.ssl=false\
 -Dcom.sun.management.jmxremote.authenticate=false\
 -Dcom.sun.management.jmxremote.port=3333 <app>

And on the local end I set up the visualvm connection as per this LINK to VisualVm help page.

With the JMX connection, I set the machine name and the port by right clicking the host on visualvm and fillng in the host port when it prompts me. At this point the status bar at the bottom shows visualvm is trying to connect to the remote machine and after a couple of minutes it fails.

With jstatd connection, I expected the remote processes to show up automatically. This didn't happen ... there were not error messages or anything.

On the socks side the following messages repeat cyclically:

debug1: Connection to port 9696 forwarding to socks port 0 requested.
debug1: channel 2: new [dynamic-tcpip]
channel 3: open failed: connect failed: Connection timed out
debug1: channel 3: free: direct-tcpip: listening port 9696 for 50.16.35.69 port 3333, connect from 127.0.0.1 port 43909, nchannels 4
debug1: Connection to port 9696 forwarding to socks port 0 requested.
debug1: channel 3: new [dynamic-tcpip]
channel 2: open failed: connect failed: Connection timed out
debug1: channel 2: free: direct-tcpip: listening port 9696 for 50.16.35.69 port 1099, connect from 127.0.0.1 port 44644, nchannels 4

I wonder if this has to do with the socks channel. I have tunnelled single ports to the remote machine before, so I cant imagine it being a configuration issue for tunneling. I'm running ubuntu linux on both sides.

Thanks for reading :)

Answer 1

JMX/RMI is hard to tunnel directly because if RMI. Basically the server creates an RMI stub definition which is armed with directions to connect back to the server from whence it came, but when you're tunelling, the stubs come down from the server, but their directions are all wrong, and they can't get there from here.

The waaay easiest way to resolve this is to ditch the RMI connector and use JMXMP. The underlying protocol is pure sockets so it's perfectly adapted for tunneling.

Answer 2

Use SSH tunnel with SOCKS proxy. See this post for more details.

Answer 3

I just did this two minutes ago ...

• Remote server with jmx on port 8686 - this port is locked down
• sshd_config on the remote box should have 'AllowTcpForwarding yes' or commented out
• Open a putty ssh session with port forwarding local port 8686 (or whatever) to localhost:8686
• Add a local jmx connection in VisualVm pointing at your local forwarded port
• VisualVm automagically sees the jvm and starts monitoring

Answer 4

Here are the steps that worked for me:

1. Launch an ejstatd in your remote host this way (in ejstatd folder): mvn exec:java -Dexec.args="-pr 2000 -ph 2001 -pv 2002" (used for "jstatd" type connection)
2. Launch your Java application with those additional Java parameters: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=2003 -Dcom.sun.management.jmxremote.rmi.port=2003 (used for "JMX" type connection)
3. Open an SSH session to the remote host tunneling those 4 ports (from 2000 to 2003 included). For example with OpenSSH client, you must add those parameters: -L2000:localhost:2000 -L2001:localhost:2001 -L2002:localhost:2002 -L2003:localhost:2003
4. Launch JVisualVM
   1. Right-click on "Local" > "Add jstatd Connection..." > "Add Custom" and enter "2000" in "Port" selection;
   2. Right-click on "Local" > "Add JMX Connection..." and enter "localhost:2003" in "Connection" input, and check "Do not require SSL connection"
   3. Your Java process will appear twice: one from the "jstatd" connection type, and one from the "JMX" connection type.

Disclaimer: I'm the author of the open source ejstatd tool.