Custom metrics adapter spewing errors "apiserver was unable to write a fallback JSON response: http2: stream closed"

[red8888]

GKE version: 1.23.14-gke.1800
metrics adapter version: gcr.io/gke-release/custom-metrics-stackdriver-adapter:v0.12.2-gke.0

Adapter was installed via:
https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/deploy/production/adapter_new_resource_model.yaml

This doesn't throw an error: kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1"

But its spewing these errors:

I saw in some github issues people saying it was lack of resources but when I look at the metrics it seems well under utilized:

I don't think its failing to read metrics, but these errors are filling up my logs and they concern me.

[5imun]

@red8888 I had the same issue, I had to deploy adapter_new_resource_model.yaml because I wanted to use external metrics but in this file along with v1beta1.external.metrics.k8s.io service you also get v1beta1.custom.metrics.k8s.io and v1beta2.custom.metrics.k8s.io.

Since I didn't use any custom metrics in the cluster so I just removed v1beta1.custom.metrics.k8s.io and v1beta2.custom.metrics.k8s.io APIServices and now I don't get these logs (this obviously won't work if you need custom metrics for scaling)

[4wdonny]

Ran into this as well. Any advice on a solution would be welcome. our Pubsub autoscalers stopped functioning and we can't tell if this message is causational or correlational

[Batuu13]

I have the same issue, any solutions?

[ajaykumar-gbi]

I have the same issue on GKE 1.25+

[ajaykumar-gbi]

@red8888 Did you find any solution for this ?

[n-oden]

FWIW I was able to work around this in the same way @5imun did -- I removed the APIService registrations for custom metrics. But this is obviously not a great solution. :(

[bigbitbus]

Same issue - cant use custom metrics for hpa autoscaling in GKE because of this.

[sosimon]

Same issue here: #318

[bigbitbus]

We have the same issue. Fortunately we wanted Pubsub-based autoscaling so that is an external-metric, so the solution that @5imun suggested worked for us.

[didlawowo]

should be a good idea to understand why we have those message

[CatherineF-dev]

Update: this PR only fixes OOM and this spam log has not been fixed.

---

Hi, could you try v0.14.2? It contains the fix #623