pushing metrics to prometheus pushgateway response 400

[zhangcughnu]

Bug Report

What did you do?
After springboot is upgraded from 2.3.4 to 2.7.18, 400 metric is pushed. The metric is normal before SpringBoot is upgraded, and the following error logs are reported:

• Pushgateway version:

    pushgateway version : 1.4.3
  

• Java dependency
  spring-boot-starter-actuator : 2.7.18
  micrometer-registry-prometheus : 1.9.17
  simpleclient_pushgateway : 0.16.0

• Logs:
  java excpetion log:
  resonse code from http://**/metrics/job/ . was 400, response body: pushed metrics are invalid or inconsistent with existing metrics : 15 erros occurred:

• collected metric "jvm_classes_unloaded_classes_total" {lable:<name:"instance" value:"" lable:<name:"job" value:"**"> counter:value:0} is not a untyped.

[beorn7]

Guessing from the provided information, it looks like the type of the pushed metrics has changed when you upgraded springboot. I assume previously jvm_classes_unloaded_classes_total was marked as an "untyped" metric, but now it's probably marked as a counter (which it should be). You run into a collision, maybe because you have pushed a metric with the same name under another grouping key.

You have to make sure that all pushers to the same Pushgateway use consistent types for metrics with the same name, and once that is the case, you have to delete all metrics with the obsolete type from the Pushgateway. Then, future pushes will succeed.