-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Info and StateSet types cause errors when scrapped by Prometheus #2248
Comments
Could you provide detailed steps on reproducing this issue? You can follow #2223 (comment) |
The problem seem to be that when Prometheus has kube-state-metrics/pkg/metricshandler/metrics_handler.go Lines 191 to 194 in 240cffd
However, there doesn't seem to be any mechanism in custom-state-metrics to revert back to Prometheus types, so we continue to expose OpenMetrics types whilst the content-type is text. I think we should rework #1930 to enable the OpenMetrics types only when the OpenMetrics format is negociated. |
Yes we can definitely improve the negotiation part and contributions that do that are welcome. |
/assign @rexagod |
IIUC All unrecognized types are categorized as COUNTERs (also, here), which are validated later on when consuming them in As discussed with Damien in a call earlier, the best way forward to bypass the absence of certain EDIT: This detect-then-enforce approach would keep things stable for dashboards, etc., that have one-or-more use-cases for the I'm wondering we might want to expose the |
Ah, So essentially, if the requested format is text-based, do nothing, else |
Fallback to `gauge` metric type when the negotiated content-type is `protofmt`-based, since Prometheus' protobuf machinery does not recognize all OpenMetrics types (`info` and `statesets` in this context). Fixes: kubernetes#2248 Signed-off-by: Pranshu Srivastava <[email protected]>
Fallback to `gauge` metric type when the negotiated content-type is `protofmt`-based, since Prometheus' protobuf machinery does not recognize all OpenMetrics types (`info` and `statesets` in this context). Fixes: kubernetes#2248 Signed-off-by: Pranshu Srivastava <[email protected]>
Fallback to `gauge` metric type when the negotiated content-type is `protofmt`-based, since Prometheus' protobuf machinery does not recognize all OpenMetrics types (`info` and `statesets` in this context). Fixes: kubernetes#2248 Signed-off-by: Pranshu Srivastava <[email protected]>
What happened:
When kube-state-metrics is configured with custom resources that produce Info and/or StateSet type of metrics, and Prometheus is configured with
--enable-feature=native-histograms
, prometheus is unable to scrape the kube-state-metrics target.Filed an issue with prometheus here prometheus/prometheus#13172 but seems like a possible issue with KSM as well.
What you expected to happen:
Successful scraping of metrics.
How to reproduce it (as minimally and precisely as possible):
Run prometheus with
--enable-feature=native-histograms
and setup kube-state-metrics with a metric of type: Info or Stateset.See a fluxcd/flux2-monitoring-example#14 or defenestration/flux2-monitoring-example#1 for an example setup.
Anything else we need to know?:
With the
native-histograms
feature disabled, prometheus seems to be able to scrape successfully. So a bug is also filed with prometheus.I guess the suggestion would be to include a setting on the custom resource to produce a promehteus-compatible type on the /metrics page for these custom resource types.
Environment:
kubectl version
): v1.27.7+k3s2The text was updated successfully, but these errors were encountered: