Custom Metrics Adapter Applicability to GKE Autopilot

[StevenACoffman]

In the tutorial https://cloud.google.com/kubernetes-engine/docs/tutorials/autoscaling-metrics#pubsub it is unclear whether the steps are applicable to an GKE Autopilot cluster.

For instance, is it necessary to install the custom-metrics-stackdriver-adapter in an GKE Autopilot cluster?

kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/deploy/production/adapter_new_resource_model.yaml

[StevenACoffman]

While I can see that in my GKE Autopilot clusters the event-exporter-gke pods are running in the kube-system namespace, there does not appear to be a pre-installed custom-metrics-stackdriver-adapter.

When installing in GKE Autopilot, you do not need need to, and in fact you cannot use node iam.gke.io/gke-metadata-server-enabled: "true". Otherwise the directions for how to Use Workload Identity are correct

See #315 and GoogleCloudPlatform/kubernetes-engine-samples#140

gcloud iam service-accounts create custom-metrics-sd-adapter --project "$GCP_PROJECT_ID"

gcloud projects add-iam-policy-binding "$GCP_PROJECT_ID" \
  --member "serviceAccount:custom-metrics-sd-adapter@$GCP_PROJECT_ID.iam.gserviceaccount.com" \
  --role "roles/monitoring.editor"

gcloud iam service-accounts add-iam-policy-binding \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:$GCP_PROJECT_ID.svc.id.goog[custom-metrics/custom-metrics-stackdriver-adapter]" \
  "custom-metrics-sd-adapter@$GCP_PROJECT_ID.iam.gserviceaccount.com"

kubectl create -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/deploy/production/adapter.yaml

kubectl annotate serviceaccount custom-metrics-stackdriver-adapter \
  "iam.gke.io/gcp-service-account=custom-metrics-sd-adapter@$GCP_PROJECT_ID.iam.gserviceaccount.com" \
  --namespace custom-metrics

[peter-volkov]

same problem.
clearly Autopilot cluster supports the external metrics from Cloud Monitoring -- it even has UI dialog to set them.
Though when you try to create a HPA object using the official example
https://cloud.google.com/kubernetes-engine/docs/tutorials/autoscaling-metrics#pubsub

it fails:

$ kubectl describe horizontalpodautoscalers.autoscaling 
Name:                                                                                    myservice
Namespace:                                                                               default
Labels:                                                                                  <none>
Annotations:                                                                             <none>
CreationTimestamp:                                                                       Mon, 12 Jun 2023 18:50:12 +0300
Reference:                                                                               Deployment/myservice
Metrics:                                                                                 ( current / target )
  "pubsub.googleapis.com|subscription|num_undelivered_messages" (target average value):  <unknown> / 2
Min replicas:                                                                            1
Max replicas:                                                                            5
Deployment pods:                                                                         1 current / 1 desired
Conditions:
  Type            Status  Reason                   Message
  ----            ------  ------                   -------
  AbleToScale     True    ReadyForNewScale         recommended size matches current size
  ScalingActive   False   FailedGetExternalMetric  the HPA was unable to compute the replica count: unable to get external metric default/pubsub.googleapis.com|subscription|num_undelivered_messages/nil: unable to fetch metrics from external metrics API: the server could not find the requested resource (get pubsub.googleapis.com|subscription|num_undelivered_messages.external.metrics.k8s.io)
  ScalingLimited  False   DesiredWithinRange       the desired count is within the acceptable range
Events:
  Type     Reason                   Age   From                       Message
  ----     ------                   ----  ----                       -------
  Warning  FailedGetExternalMetric  1s    horizontal-pod-autoscaler  unable to get external metric default/pubsub.googleapis.com|subscription|num_undelivered_messages/nil: unable to fetch metrics from external metrics API: the server could not find the requested resource (get pubsub.googleapis.com|subscription|num_undelivered_messages.external.metrics.k8s.io)

I'm pretty sure that the metric exists -- I can see it in Cloud Monitoring

[peter-volkov]

ok. it works after installing custom metric adapter. I thought it should be preinstalled

kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/deploy/production/adapter_new_resource_model.yaml

[StevenACoffman]

@Shabirmean @linxiulei @CatherineF-dev Can you clarify if the custom metric adapter is now pre-installed or needs to be manually installed in recent GKE Autopilot?

The root ReadMe.md in the k8s-stackdriver repository states:

Google Cloud Operations integration for GKE

❗ Tools in this repository are not meant for
end-users.
It contains source code for tools that are pre-installed in
the GKE clusters.

[CatherineF-dev]

Needs to be manually installed.

[StevenACoffman]

Thanks! It might be nice to update the README to say something like:

Only the custom metric adapter here is meant for end-users to manually install. All other tools in this repository are not meant for end-users.