Java JMX with Docker

The purpose of this project is to present the configuration settings required to expose a JMX port from a JVM running inside a Docker container.

Docker requires ports to be declared before the application runs. This conflicts with JMX over RMI (the default JMX protocol), which relies on establishing communication using random ports negotiated at connection time. The randomly negotiated JMX ports can't be declared in the Docker config, causing JMX connections to fail.

If connecting from another container linked to the JVM container (same Docker network) then all ports will be accessible, including the randomly negotiated ones. However, the typical use case for JMX monitoring is to connect from outside the docker network (via mapping to a host port).

We get around these limitations with careful configuration of the JMX properties. The main tricks:

• set com.sun.management.jmxremote.port and com.sun.management.jmxremote.rmi.port to the exposed port, in our case 9010, and
• set com.sun.management.jmxremote.host and java.rmi.server.hostname to the catch-all IP address 0.0.0.0.

TL;DR -- entrypoint.sh

Usage

./mvnw package
docker-compose up --build

Docker Compose will start the application and expose port 9010 as a JMX port on the docker host.

Using jconsole or VisualVM, you can connect to localhost:9010.

Notes

The goal of this configuration is to connect with a JMX/RMI client from outside of the Docker internal network, usually via a port mapped to a host port.

The RMI transport is included with the JVM, and therefore is supported by all the JMX clients (JConsole, VisualVM, etc).

Here are some considerations when setting the JVM arguments:

1. com.sun.management.jmxremote.port and com.sun.management.jmxremote.rmi.port

   These properties are set to the same value for convenience. They don't have to be the same, but you have to expose one extra port if they're not equal.

   If you don't declare the RMI port, the RMI protocol will choose a random port at connection time after the initial handshake. This will cause the JMX client to hang as the port will not be externally accessible.

2. com.sun.management.jmxremote.host

   This property is required if java.rmi.server.hostname is not set and represents the externally accessible hostname or IP of the JVM, used as part of the JmxConnectorUrl. If ConnectorBootstrap logging is enabled, the URL will be printed at JVM startup:

   CONFIG: JMX Connector ready at: service:jmx:rmi:///jndi/rmi://172.18.0.2:9010/jmxrmi

   When running in Docker this hostname or IP should be externally accessible. The value is usually passed into the container through an environment variable, as Docker provides no mechanism for looking up the Docker host's hostname or IP.

   If neither this property nor java.rmi.server.hostname are set, you will get this error at JVM startup:

   Error: Exception thrown by the agent : java.net.MalformedURLException: Cannot give port number without host name

   In our case, we set the host to 0.0.0.0 for the JVM to listen on any available interface.

3. java.util.logging.config.file

   The optional path to a logging.properties file that configures the Java Logging framework to print RMI debugging messages.

   Example logging output:

   Mar 23, 2017 8:56:26 AM ConnectorBootstrap startRemoteConnectorServer FINEST: Starting JMX Connector Server: com.sun.management.jmxremote.port=9010 com.sun.management.jmxremote.host=0.0.0.0 com.sun.management.jmxremote.rmi.port=9010 com.sun.management.jmxremote.ssl=false com.sun.management.jmxremote.registry.ssl=false com.sun.management.jmxremote.ssl.config.file=null com.sun.management.jmxremote.ssl.enabled.cipher.suites=null com.sun.management.jmxremote.ssl.enabled.protocols=null com.sun.management.jmxremote.ssl.need.client.auth=false com.sun.management.jmxremote.authenticate=false No Authentication Mar 23, 2017 8:56:26 AM ConnectorBootstrap startRemoteConnectorServer CONFIG: JMX Connector ready at: service:jmx:rmi:///jndi/rmi://0.0.0.0:9010/jmxrmi

   JMX logging configuration happens early in the JVM startup and uses the Java Logging framework. This logging is useful for debugging purposes.

4. com.sun.management.config.file

   This optional configuration option points to a file that is read in by ConnectorBootstrap at startup time to set com.sun.management.jmxremote.* properties. However, since no environment variable substitution is done any properties that must be set via environment variables cannot be specified in that file, and must be passed from this shell script (see below).

   The properties in the management.properties file can be passed directly to the JVM as command line arguments. See entrypoint.sh.

5. java.rmi.server.hostname

   This is a critical property when using JMX with a JVM running inside a Docker container. It should be set to the externally accessible hostname or IP of the Docker container, same as com.sun.management.jmxremote.host.

   If this property is incorrect (or not set) all JMX connections will fail!

   In our case, we use the catch-all IP 0.0.0.0 to have the JVM listen on any available address. This avoids us having to specify the host IP of the Docker machine, and requires no further special configuration.

Links

GitHub

• oracle/docker-images - Very informative, talks about JMXMP too.
• nolexa/docker-jmx-demo - Explains the same thing as this repository.
• gimerstedt/jmx-to-spring-boot-in-docker - Spring Boot centric
• zacker330/Java-JMX-example - adds simple authentication, Jolokia agent
• jolokia/jolokia - a JVM agent that uses JSON over HTTP, instead of RMI, has other features
• prometheus/jmx_exporter - a JVM agent that exposes JMX metrics for Prometheus

Blog Posts

• JMX Ports (Baeldung) - for local apps not Docker, also describes -XX:+DisableAttachMechanism and com.sun.management.jmxremote
• Remote Java Debugging With Docker
• Monitoring Java Applications Running Inside Docker Containers
• Monitoring JVM apps in a Docker environment
• JMX Monitoring with Docker and the ELK Stack
• How to connect VisualVM to Docker
• JMX: RMI vs. JMXMP - Outdated, but talks about why JMXMP fits better with Docker

Forums

• https://forums.docker.com/t/enable-jmx-rmi-access-to-a-docker-container/625/5

StackOverflow

• Why Java opens 3 ports when JMX is configured?
• How to activate JMX on my JVM for access with jconsole?
• How to access JMX interface in docker from outside?
• Access through jmx to java application into Docker container on remote host in local network - Very concise explanation of using JMXMP
• multiple app nodes how to expose jmx in kubernetes?
• Profiling Java application in kubernetes
• JConsole over ssh local port forwarding
• How to use VisualVM and JMX?

YouTube

• Using JMX to monitor an Application Server running on Docker - This video sets java.rmi.server.hostname to the Docker host IP address, which is not what we recommend in this repo. It's also using some specific Dell JMX web interface. Consider removing this link.

Other documentation

• confluent
• Oracle Guide