How can I connect to JMX through Kubernetes managed Docker containers?

Question

I am trying to connect to a JMX port that I have defined to be 1099 (as is the default) on my Java application (Java 8 and with Spring Boot 1.4 and Tomcat) using a client such as JConsole, Java Mission Control, or Java VisualVM, but I am reaching an error:

java.rmi.ConnectException: Connection refused to host: 10.xxx.xxx.xx, nested exception is:
     java.net.ConnectException: Connection timed out
     ...

Note that I hid the exact host IP, but it is the pod IP of the particular Kubernetes-managed Docker container that my service is deployed in. I try to connect to the JMX port using the following service URL:

jconsole service:jmx:rmi://<nodeIP>:<nodePort>/jndi/rmi://<nodeIP>:<nodePort>/jmxrmi

I know that JMX opens a random high port, and I have tried to resolve that by including a custom @Configuration class that forces that high port to also serve on port 1099. I have gone into the actual container and pod and ran

netstat -tulpn

to see the ports that are opened, and I have confirmed that the only ports opened are 8443 (which my application is running on) and 1099 (the JMX port); this indicates that my class works. I also ensured that port 1099 is open on the Kubernetes side, so that is not what is blocking it.

As many answers surrounding a JMX remote connection have suggested, I have tried many variations of the following Java options to no avail:

-Dcom.sun.management.jmxremote  -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.jmi.port=1099 -Djava.rmi.server.hostname=0.0.0.0

The answers have suggested forcing the JMX port and the RMI registry port to be the same, but none of these have worked.

I believe the issue may be because the hostname (as I tried to make dynamic with -Djava.rmi.server.hostname=0.0.0.0) can not resolve to the different hostnames (pod IPs) that are created every time my service is deployed.

Thus, it looks like the connection cannot complete because JMX is unable to see what hostname Kubernetes is assigning after my service is deployed.

Is there a way for JMX to recognize the Kubernetes hostname? Or, is there some other way to connect remotely to a JMX port through a Kubernetes-deployed service?

EDIT 1: I have done some additional research, and maybe an optional JMXMP instead of RMI may work? Has anyone gotten this working with Tomcat?

have you tried using kubectl port-forward [podname] 1099:1099 to test if it's listening? — 3ocene
Also, the 10.xxx.xxx.xxx range of IPs are reserved. That isn't a public IP so you likely can't connect to it. You probably need to set up a service and an ingress to talk in to your pod via the node's IP address. — 3ocene
@3ocene port-forward is not an option in our case. We need to be able to use a node-port for our situation. The 10.xxx.xxx.xxx address is a private IP and the cluster is part of our private network. Note that we did a ssh-tunnel from a node in the cluster to the podIP:1099 and the connection work but it doesn't work with the node:nodePort. — realgenob
Just to be clear, @realgenob is also working with me and experiencing this same issue. — Tremaine
Have you tried specifying -Djava.rmi.server.hostname=10.x.x.x? We had a similar setup with Openstack VMs, where OpenStack assigns internal IPs to the VMs, but you use external routed IPs to access the applications. So we specified that parameter with the external IP and it worked (of course then you will have the next problem when you want to make it dynamically). — dunni

Sergiu Vidrascu Sergiu Vidrascu · Accepted Answer · 2016-12-10T09:36:18

The, jmx remote connection is a pain to work with, proxying it is impossible from my view. I had similar problems and in the end I just used jolokia to connect.

Jolokia is a JMX-HTTP bridge giving an alternative to JSR-160 connectors. It is an agent based approach with support for many platforms. In addition to basic JMX operations it enhances JMX remoting with unique features like bulk requests and fine grained security policies. -> http://jolokia.org

How can I connect to JMX through Kubernetes managed Docker containers?

1 Answers