custom-metrics-sd-adapter: 403s when fetching metrics from another project

[beyondbill]

I got the following error when using the credentials of the GCP service account from a project different than where custom-metrics-stackdriver-adapter runs.

provider.go:271] Failed request to stackdriver api: googleapi: Error 403: Permission monitoring.metricDescriptors.list denied (or the resource may not exist)., forbidden

I've confirmed that GOOGLE_APPLICATION_CREDENTIALS is set correctly in the container and the service account has enough permissions to read metrics. The way I validated them was to swap the custom-metrics-stackdriver-adapter image with google/cloud-sdk and successfully got a list of metrics back from within an interactive shell in the container

# metrics-and-alarms is the project ID where metrics live

curl -H "Authorization: Bearer \"$(gcloud auth application-default print-access-token)\"" https://monitoring.googleapis.com/v3/projects/metrics-and-alarms/metricDescriptors

Here's the Kubernetes template I use. Did I mis-config anything?

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/instance: custom-metrics-stackdriver-adapter-a1226
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: custom-metrics-stackdriver-adapter
    app.kubernetes.io/version: v0.12.2-gke.0
    argocd.argoproj.io/instance: custom-metrics-stackdriver-adapter-a1226
    helm.sh/chart: custom-metrics-stackdriver-adapter-0.2.0
  name: custom-metrics-stackdriver-adapter-a1226
  namespace: custom-metrics
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/instance: custom-metrics-stackdriver-adapter-a1226
      app.kubernetes.io/name: custom-metrics-stackdriver-adapter
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: custom-metrics-stackdriver-adapter-a1226
        app.kubernetes.io/name: custom-metrics-stackdriver-adapter
    spec:
      containers:
        - command:
            - /adapter
            - '--use-new-resource-model=true'
            - '--fallback-for-container-metrics=true'
          env:
            - name: GOOGLE_APPLICATION_CREDENTIALS
              value: /etc/creds/token.json
          image: 'gcr.io/gke-release/custom-metrics-stackdriver-adapter:v0.12.2-gke.0'
          imagePullPolicy: IfNotPresent
          name: custom-metrics-stackdriver-adapter
          volumeMounts:
            - mountPath: /etc/creds
              name: gcp-service-account-creds
              readOnly: true
      serviceAccountName: custom-metrics-stackdriver-adapter-a1226
      volumes:
        - name: gcp-service-account-creds
          secret:
            defaultMode: 420
            secretName: custom-metrics-stackdriver-adapter