fix: fallback to gauge for protofmt-based negotiations

[rexagod]

What this PR does / why we need it: Fallback to gauge metric type when the negotiated content-type is protofmt-based, since Prometheus' protobuf machinery does not recognize all OpenMetrics types (info and statesets in this context).

How does this change affect the cardinality of KSM: None.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #2248

[rexagod]

Need to test this.

[rexagod]

Verified using the following configurations. Error logs were reported without this patch, but none when this was applied.

Details

Prometheus error log

ts=2023-12-13T16:01:02.547Z caller=scrape.go:1418 level=debug component="scrape manager" scrape_pool=kube-state-metrics target=http://127.0.0.1:8080/metrics msg="Append failed" err="invalid metric type \"info\""

KSM CRS Configuration

	  ...
      metrics:
        - name: "foo_info"
          help: "foo_help"
          each:
            type: Info
            info:
              path: [metadata]
              labelsFromPath:
                "name": [name]

Prometheus Configuration

global:
  evaluation_interval: 15s
  scrape_interval:     15s
  scrape_protocols: [PrometheusProto]
scrape_configs:
  - job_name: 'kube-state-metrics'
    static_configs:
      - targets: ['127.0.0.1:8080']

Generated CRS Metric (from Prometheus UI)

kube_customresource_foo_info{customresource_group="contoso.com", customresource_kind="MyPlatform", customresource_version="v1alpha1", instance="127.0.0.1:8080", job="kube-state-metrics", name="test-dotnet-app"} 1

[logicalhan]

/triage accepted
/assign @dgrisonnet

[rexagod]

@dgrisonnet Yes, I believe we talked about dropping metric-standard lock-ins, and implementing a one-fits-all mechanism (probably on top of gauges) that allows us to generate an exposition format that is ingestible by most metric backends, if not all.

[rexagod]

(friendly ping @mrueg @dgrisonnet to /lgtm)

[remram44]

2.11.0 introduced a regression, as metrics include stateset entries which are refused by Prometheus. This worked correctly on 2.10 so I'm stuck until this MR is merged.

[dgrisonnet]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dgrisonnet, mrueg, rexagod

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [dgrisonnet,mrueg,rexagod]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[dgrisonnet]

/unhold

[Efrat19]

Hey @rexagod ! is this also meant to fix #1973 replicating in 2.110??

[rexagod]

It should, yes.

[LaikaN57]

Hi folks, is this change targeted to be in the release-2.11 cut (which has the k8s 1.28 support of the client libs)?

@dgrisonnet Looks like this fix got merged in too late for v2.11.0. How do we feel about cutting a new release soon™ to get people unblocked? (Thanks in advance.)

[LaikaN57]

Hi folks, is this change targeted to be in the release-2.11 cut (which has the k8s 1.28 support of the client libs)?

@dgrisonnet Looks like this fix got merged in too late for v2.11.0. How do we feel about cutting a new release soon™ to get people unblocked? (Thanks in advance.)

@rexagod bump ^

[rexagod]

I'll put #2335 up for review today.

[rexagod]

Sharing here for visibility: #2335 (comment).

[seruman]

hi @rexagod,

i'm probably missing something but should not it also fallback to gauge on text/plain?

with --enable-feature=native-histograms prometheus (2.52.0) sends;

Accept: application/vnd.google.protobuf;proto=io.prometheus.client.MetricFamily;encoding=delimited;q=0.5,application/openmetrics-text;version=1.0.0;q=0.4,application/openmetrics-text;version=0.0.1;q=0.3,text/plain;version=0.0.4;q=0.2,*/*;q=0.1

but handler falls-back to text/plain here;

kube-state-metrics/pkg/metricshandler/metrics_handler.go

Lines 200 to 206 in d3f0c18

	contentType := expfmt.NegotiateIncludingOpenMetrics(r.Header)
	// We do not support protobuf at the moment. Fall back to FmtText if the negotiated exposition format is not FmtOpenMetrics See: https://github.com/kubernetes/kube-state-metrics/issues/2022.
	if contentType.FormatType() != expfmt.TypeOpenMetrics {
	contentType = expfmt.NewFormat(expfmt.TypeTextPlain)
	}

thus exporter responds with type info that makes prometheus unhappy ; invalid metric type "info".