Spring Boot 3.x Micrometer StatsD registry does not send metrics to UDP port when another StatsD client is active on the same port

[spachari-im]

Describe the bug
In Spring Boot 3.x, Micrometer's inbuilt StatsD agent does not report JMX metrics when another StatsDClient is created to send custom metrics to the same host & port.

Environment
Spring Boot 3.2.7, Spring Camel version 4.5.0 application.

• Micrometer version [1.12.7]
• Micrometer registry - StatsD [1.9.2]
• OS: MacOS
• Java version: 17

To Reproduce
How to reproduce the bug:

• Declare a StatsDClient bean to send custom metrics to Datadog

StatsDClient statsDClient = new NonBlockingStatsDClientBuilder()
                    .prefix(PREFIX)
                    .hostname(localhost)
                    .port(8125)
                    .build(); 

• Add mcrometer-registry-statsd to project (in pom.xml).
• Enable micrometer statsd metrics.

management.statsd.metrics.export.enabled=true
management.statsd.metrics.export.flavor=datadog
management.statsd.metrics.export.protocol=udp
management.statsd.metrics.export.host=localhost
management.statsd.metrics.export.port=8125
management.statsd.metrics.export.polling-frequency=10s
management.statsd.metrics.export.publish-unchanged-meters=true

Note: the Micrometer StatsD client and the custom StatsDClient both send metrics to the same host and port

• The application does not report the implicit JVM metrics sent by Micrometer to the StatsD Agent to port 8125 (netcat -ulp 8125).
• Only the custom metrics sent by the Datadog StatsD client is being reported to port 8125.
• When the Datadog StatsD client is removed, the JVM metrics are successfully reported to port 8125.

Expected behavior

• On port 8125 (netcat -ulp 8125) both the custom metrics sent through Datadog StatsD client and the internal JVM metrics sent by Micormeter should be visible.
• Micrometer should report some warning in logs if metrics cannot be sent to the UDP port, instead of ignoring the metric.

Additional context

• This configuration runs successfully on Spring Boot 2.x and Micrometer 1.9.2.
• When the Datadog StatsD client is removed, the JVM metrics are successfully reported to port 8125.
• It appears that the Micrometer StatsD agent does not work if there is another StatsD client sending to the same port (I am not sure of this)
• There are no warnings in the logs to give an idea of why this happens.

[shakuzen]

Thank you for the report, and sorry for the late triage on it.

• Micrometer version [1.12.7]
• Micrometer registry - StatsD [1.9.2]

You should use a single, consistent version of Micrometer. If you are using Spring Boot's dependency management, it should manage the version for you so you don't end up with two different versions like this. Using different versions has potential to cause issues.

I tried to reproduce the issue and I was not able to do so. I added the Dogstatsd client (StatsDClient) to a Spring Boot 3.4.0 app with Micrometer's statsd registry, and all metrics (from each client) were available in Datadog. Could you please try with the latest versions and ensure consistent versions are used to see if this is still an issue for you?

[spachari-im]

@shakuzen thanks for triaging this.

I did not override the micrometer version. I just specified it in the ticket. The only micrometer dependency in the propject's pom file is this:

<dependency>
      <groupId>io.micrometer</groupId>
      <artifactId>micrometer-registry-statsd</artifactId>
    </dependency>

I tried this test after upgrading the Spring Boot version to 3.4.0. I get the same behavior - if a StatsD client is added for custom metrics, the custom metrics are reported but JVM metrics are not shown.

The JVM metrics are shown when the StatsD client is removed.

I think this is because of a setting enabled in the properties file or a dependency related to the Camel starter dependency - org.apache.camel.springboot.

Can you please post all the code you have used (properties enabled, bean declaration and pom dependency)?