external.metrics api does not goes up. only custom.metrics api. ='(

[pedropaulopereiraamarante]

What happened?:
I would like to be able to get the metrics from both APIs.

$kubectl get --raw /apis/external.metrics.k8s.io/v1beta1 | jq
Error from server (NotFound): the server could not find the requested resource

$kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "pods/sidekiq_enqueued_jobs",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "namespaces/sidekiq_enqueued_jobs",
      "singularName": "",
      "namespaced": false,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    }
  ]
}

What did you expect to happen?:
I should be able to get metrics from both APIs (external/custom). I've tried everything, and only the custom API is working.
Please provide the prometheus-adapter config:

I have tried all possible configurations, none result in the external.metrics API working.

data:
  config.yaml: |
    rules:
      - seriesQuery: 'sidekiq_queue_enqueued_jobs{name="campaign-notification-events",namespace!="",pod!=""}'
        resources:
          overrides:
            namespace: {resource: "namespace"}
            pod: {resource: "pod"}
        name:
          matches: "^sidekiq_queue_enqueued_jobs$"
          as: "sidekiq_enqueued_jobs"
        metricsQuery: |
          topk(1, sidekiq_queue_enqueued_jobs{
            name="campaign-notification-events",
            <<.LabelMatchers>>
          })
  rules.external: |
    rules:
      externalRules:
      - seriesQuery: '{__name__=~"^sidekiq_queue_enqueued_jobs$"}'
        resources:
          overrides:
            namespace: {resource: "namespace"}
        name:
          matches: "^sidekiq_queue_enqueued_jobs$"
          as: "sidekiq_enqueued_jobs_namespace"
        metricsQuery: 'sum(sidekiq_queue_enqueued_jobs{<<.LabelMatchers>>}) by (namespace)'

ex: 2 not worked too.**

data:
  config.yaml: |
    rules:
      - seriesQuery: 'sidekiq_queue_enqueued_jobs{name="campaign-notification-events",namespace!="",pod!=""}'
        resources:
          overrides:
            namespace: {resource: "namespace"}
            pod: {resource: "pod"}
        name:
          matches: "^sidekiq_queue_enqueued_jobs$"
          as: "sidekiq_enqueued_jobs"
        metricsQuery: |
          topk(1, sidekiq_queue_enqueued_jobs{
            name="campaign-notification-events",
            <<.LabelMatchers>>
          })

    externalRules:
      - seriesQuery: '{__name__=~"^sidekiq_queue_enqueued_jobs$"}'
        resources:
          overrides:
            namespace: {resource: "namespace"}
        name:
          matches: "^sidekiq_queue_enqueued_jobs$"
          as: "sidekiq_enqueued_jobs_namespace"
        metricsQuery: 'sum(sidekiq_queue_enqueued_jobs{<<.LabelMatchers>>}) by (namespace)'

prometheus-adapter config

Please provide the HPA resource used for autoscaling:

HPA yaml

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: growth-service-campaign-notification-sidekiq-hpa
  namespace: growth-service
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: growth-service-campaign-notification-sidekiq
  minReplicas: 5
  maxReplicas: 21
  metrics:
    - type: External
      external:
        metric:
          name: sidekiq_enqueued_jobs
        target:
          type: AverageValue
          averageValue: 900k

Please provide the HPA status:

Name:                                                        growth-service-campaign-notification-sidekiq-hpa
Namespace:                                                   growth-service
Labels:                                                      <none>
Annotations:                                                 <none>
CreationTimestamp:                                           Tue, 18 Jun 2024 17:44:48 -0300
Reference:                                                   Deployment/growth-service-campaign-notification-sidekiq
Metrics:                                                     ( current / target )
  "sidekiq_enqueued_jobs_namespace" (target average value):  <unknown> / 900k
Min replicas:                                                5
Max replicas:                                                21
Deployment pods:                                             20 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetExternalMetric  the HPA was unable to compute the replica count: unable to get external metric growth-service/sidekiq_enqueued_jobs_namespace/nil: unable to fetch metrics from external metrics API: the server could not find the requested resource (get sidekiq_enqueued_jobs_namespace.external.metrics.k8s.io)
Events:
  Type     Reason                        Age               From                       Message
  ----     ------                        ----              ----                       -------
  Warning  FailedGetExternalMetric       1s (x2 over 16s)  horizontal-pod-autoscaler  unable to get external metric growth-service/sidekiq_enqueued_jobs_namespace/nil: unable to fetch metrics from external metrics API: the server could not find the requested resource (get sidekiq_enqueued_jobs_namespace.external.metrics.k8s.io)
  Warning  FailedComputeMetricsReplicas  1s (x2 over 16s)  horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get sidekiq_enqueued_jobs_namespace external metric value: failed to get sidekiq_enqueued_jobs_namespace external metric: unable to get external metric growth-service/sidekiq_enqueued_jobs_namespace/nil: unable to fetch metrics from external metrics API: the server could not find the requested resource (get sidekiq_enqueued_jobs_namespace.external.metrics.k8s.io)

Please provide the prometheus-adapter logs with -v=6 around the time the issue happened:

prometheus-adapter logs

I0618 20:52:33.477517 1 adapter.go:114] successfully using in-cluster auth I0618 20:52:33.484906 1 round_trippers.go:553] GET https://10.100.0.1:443/api?timeout=32s 200 OK in 6 milliseconds I0618 20:52:33.488166 1 round_trippers.go:553] GET https://10.100.0.1:443/apis?timeout=32s 200 OK in 2 milliseconds I0618 20:52:33.504567 1 api.go:88] GET http://prometheus.monitoring.svc:9090/api/v1/series?match%5B%5D=sidekiq_queue_enqueued_jobs%7Bname%3D%22campaign-notification-events%22%2Cnamespace%21%3D%22%22%2Cpod%21%3D%22%22%7D&start=1718743893.491 200 OK I0618 20:52:33.794448 1 serving.go:342] Generated self-signed cert (/tmp/cert/apiserver.crt, /tmp/cert/apiserver.key) I0618 20:52:33.795538 1 dynamic_serving_content.go:113] "Loaded a new cert/key pair" name="serving-cert::/tmp/cert/apiserver.crt::/tmp/cert/apiserver.key" I0618 20:52:34.366098 1 round_trippers.go:553] GET https://10.100.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication 200 OK in 4 milliseconds I0618 20:52:34.366900 1 requestheader_controller.go:244] Loaded a new request header values for RequestHeaderAuthRequestController I0618 20:52:34.367302 1 maxinflight.go:140] "Initialized nonMutatingChan" len=400 I0618 20:52:34.367360 1 maxinflight.go:146] "Initialized mutatingChan" len=200 I0618 20:52:34.367407 1 timing_ratio_histogram.go:203] "TimingRatioHistogramVec.NewForLabelValuesSafe hit the inefficient case" fqName="apiserver_flowcontrol_read_vs_write_current_requests" labelValues=[executing readOnly] I0618 20:52:34.367451 1 timing_ratio_histogram.go:203] "TimingRatioHistogramVec.NewForLabelValuesSafe hit the inefficient case" fqName="apiserver_flowcontrol_read_vs_write_current_requests" labelValues=[executing mutating] I0618 20:52:34.367488 1 maxinflight.go:117] "Set denominator for readonly requests" limit=400 I0618 20:52:34.367512 1 maxinflight.go:121] "Set denominator for mutating requests" limit=200 I0618 20:52:34.369387 1 config.go:814] Not requested to run hook priority-and-fairness-config-consumer I0618 20:52:34.374917 1 healthz.go:176] Installing health checkers for (/healthz): "ping","log","poststarthook/max-in-flight-filter","poststarthook/storage-object-count-tracker-hook" I0618 20:52:34.375229 1 healthz.go:176] Installing health checkers for (/livez): "ping","log","poststarthook/max-in-flight-filter","poststarthook/storage-object-count-tracker-hook" I0618 20:52:34.375483 1 healthz.go:176] Installing health checkers for (/readyz): "ping","log","poststarthook/max-in-flight-filter","poststarthook/storage-object-count-tracker-hook","shutdown" I0618 20:52:34.375965 1 genericapiserver.go:524] MuxAndDiscoveryComplete has all endpoints registered and discovery information is complete I0618 20:52:34.380054 1 round_trippers.go:553] GET https://10.100.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication 200 OK in 4 milliseconds I0618 20:52:34.384706 1 tlsconfig.go:200] "Loaded serving cert" certName="serving-cert::/tmp/cert/apiserver.crt::/tmp/cert/apiserver.key" certDetail="\"localhost@1718743953\" [serving] validServingFor=[127.0.0.1,localhost,localhost] issuer=\"localhost-ca@1718743953\" (2024-06-18 19:52:33 +0000 UTC to 2025-06-18 19:52:33 +0000 UTC (now=2024-06-18 20:52:34.384668328 +0000 UTC))" I0618 20:52:34.385141 1 named_certificates.go:53] "Loaded SNI cert" index=0 certName="self-signed loopback" certDetail="\"apiserver-loopback-client@1718743954\" [serving] validServingFor=[apiserver-loopback-client] issuer=\"apiserver-loopback-client-ca@1718743953\" (2024-06-18 19:52:33 +0000 UTC to 2025-06-18 19:52:33 +0000 UTC (now=2024-06-18 20:52:34.385114205 +0000 UTC))" I0618 20:52:34.385335 1 secure_serving.go:210] Serving securely on [::]:6443 I0618 20:52:34.385437 1 genericapiserver.go:672] [graceful-termination] waiting for shutdown to be initiated I0618 20:52:34.385550 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController I0618 20:52:34.385633 1 shared_informer.go:311] Waiting for caches to sync for RequestHeaderAuthRequestController I0618 20:52:34.385753 1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/tmp/cert/apiserver.crt::/tmp/cert/apiserver.key" I0618 20:52:34.385955 1 tlsconfig.go:240] "Starting DynamicServingCertificateController" I0618 20:52:34.386206 1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0618 20:52:34.386229 1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0618 20:52:34.386364 1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0618 20:52:34.386380 1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0618 20:52:34.386641 1 reflector.go:287] Starting reflector *v1.ConfigMap (12h0m0s) from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:231 I0618 20:52:34.386658 1 reflector.go:323] Listing and watching *v1.ConfigMap from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:231 I0618 20:52:34.387699 1 reflector.go:287] Starting reflector *v1.ConfigMap (12h0m0s) from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:231 I0618 20:52:34.387719 1 reflector.go:323] Listing and watching *v1.ConfigMap from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:231 I0618 20:52:34.387921 1 reflector.go:287] Starting reflector *v1.ConfigMap (12h0m0s) from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:231 I0618 20:52:34.388082 1 reflector.go:323] Listing and watching *v1.ConfigMap from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:231 I0618 20:52:34.388641 1 round_trippers.go:553] GET https://10.100.0.1:443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&limit=500&resourceVersion=0 200 OK in 1 milliseconds I0618 20:52:34.389215 1 round_trippers.go:553] GET https://10.100.0.1:443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&limit=500&resourceVersion=0 200 OK in 0 milliseconds I0618 20:52:34.391774 1 round_trippers.go:553] GET https://10.100.0.1:443/api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=14273812&timeout=8m21s&timeoutSeconds=501&watch=true 200 OK in 1 milliseconds I0618 20:52:34.392861 1 round_trippers.go:553] GET https://10.100.0.1:443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&limit=500&resourceVersion=0 200 OK in 2 milliseconds I0618 20:52:34.394447 1 round_trippers.go:553] GET https://10.100.0.1:443/api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=14273812&timeout=9m49s&timeoutSeconds=589&watch=true 200 OK in 0 milliseconds I0618 20:52:34.396393 1 round_trippers.go:553] GET https://10.100.0.1:443/api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=14273812&timeout=6m55s&timeoutSeconds=415&watch=true 200 OK in 0 milliseconds I0618 20:52:34.486580 1 shared_informer.go:341] caches populated I0618 20:52:34.486598 1 shared_informer.go:341] caches populated I0618 20:52:34.486603 1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0618 20:52:34.486615 1 shared_informer.go:318] Caches are synced for RequestHeaderAuthRequestController I0618 20:52:34.486748 1 shared_informer.go:341] caches populated I0618 20:52:34.486758 1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0618 20:52:34.486927 1 tlsconfig.go:178] "Loaded client CA" index=0 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"front-proxy-ca\" [] validServingFor=[front-proxy-ca] issuer=\"<self>\" (2024-05-23 13:48:40 +0000 UTC to 2034-05-21 13:53:40 +0000 UTC (now=2024-06-18 20:52:34.48690411 +0000 UTC))" I0618 20:52:34.487573 1 tlsconfig.go:200] "Loaded serving cert" certName="serving-cert::/tmp/cert/apiserver.crt::/tmp/cert/apiserver.key" certDetail="\"localhost@1718743953\" [serving] validServingFor=[127.0.0.1,localhost,localhost] issuer=\"localhost-ca@1718743953\" (2024-06-18 19:52:33 +0000 UTC to 2025-06-18 19:52:33 +0000 UTC (now=2024-06-18 20:52:34.487545323 +0000 UTC))" I0618 20:52:34.488266 1 named_certificates.go:53] "Loaded SNI cert" index=0 certName="self-signed loopback" certDetail="\"apiserver-loopback-client@1718743954\" [serving] validServingFor=[apiserver-loopback-client] issuer=\"apiserver-loopback-client-ca@1718743953\" (2024-06-18 19:52:33 +0000 UTC to 2025-06-18 19:52:33 +0000 UTC (now=2024-06-18 20:52:34.488236413 +0000 UTC))" I0618 20:52:34.488535 1 tlsconfig.go:178] "Loaded client CA" index=0 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"kubernetes\" [] validServingFor=[kubernetes] issuer=\"<self>\" (2024-05-23 13:48:52 +0000 UTC to 2034-05-21 13:53:52 +0000 UTC (now=2024-06-18 20:52:34.488516354 +0000 UTC))" I0618 20:52:34.488575 1 tlsconfig.go:178] "Loaded client CA" index=1 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"kubernetes\" [] validServingFor=[kubernetes] issuer=\"<self>\" (2024-05-23 13:54:14 +0000 UTC to 2034-05-21 13:59:14 +0000 UTC (now=2024-06-18 20:52:34.488553243 +0000 UTC))" I0618 20:52:34.488599 1 tlsconfig.go:178] "Loaded client CA" index=2 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"kubernetes\" [] validServingFor=[kubernetes] issuer=\"<self>\" (2024-06-06 16:10:31 +0000 UTC to 2034-06-04 16:15:31 +0000 UTC (now=2024-06-18 20:52:34.488582142 +0000 UTC))" I0618 20:52:34.488620 1 tlsconfig.go:178] "Loaded client CA" index=3 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"kubernetes\" [] validServingFor=[kubernetes] issuer=\"<self>\" (2024-06-06 16:10:31 +0000 UTC to 2034-06-04 16:15:31 +0000 UTC (now=2024-06-18 20:52:34.488610675 +0000 UTC))" I0618 20:52:34.488636 1 tlsconfig.go:178] "Loaded client CA" index=4 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"kubernetes\" [] validServingFor=[kubernetes] issuer=\"<self>\" (2024-06-06 16:17:00 +0000 UTC to 2034-06-04 16:22:00 +0000 UTC (now=2024-06-18 20:52:34.488626953 +0000 UTC))" I0618 20:52:34.488675 1 tlsconfig.go:178] "Loaded client CA" index=5 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"kubernetes\" [] validServingFor=[kubernetes] issuer=\"<self>\" (2024-06-11 17:02:33 +0000 UTC to 2034-06-09 17:07:33 +0000 UTC (now=2024-06-18 20:52:34.488650408 +0000 UTC))" I0618 20:52:34.488696 1 tlsconfig.go:178] "Loaded client CA" index=6 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"kubernetes\" [] validServingFor=[kubernetes] issuer=\"<self>\" (2024-06-11 17:02:32 +0000 UTC to 2034-06-09 17:07:32 +0000 UTC (now=2024-06-18 20:52:34.488686405 +0000 UTC))" I0618 20:52:34.488715 1 tlsconfig.go:178] "Loaded client CA" index=7 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"kubernetes\" [] validServingFor=[kubernetes] issuer=\"<self>\" (2024-06-11 17:08:53 +0000 UTC to 2034-06-09 17:13:53 +0000 UTC (now=2024-06-18 20:52:34.488705531 +0000 UTC))" I0618 20:52:34.488777 1 tlsconfig.go:178] "Loaded client CA" index=8 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"kubernetes\" [] validServingFor=[kubernetes] issuer=\"<self>\" (2024-06-11 18:36:09 +0000 UTC to 2034-06-09 18:41:09 +0000 UTC (now=2024-06-18 20:52:34.488729 +0000 UTC))" I0618 20:52:34.488818 1 tlsconfig.go:178] "Loaded client CA" index=9 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"kubernetes\" [] validServingFor=[kubernetes] issuer=\"<self>\" (2024-06-11 18:36:06 +0000 UTC to 2034-06-09 18:41:06 +0000 UTC (now=2024-06-18 20:52:34.488798284 +0000 UTC))" I0618 20:52:34.488881 1 tlsconfig.go:178] "Loaded client CA" index=10 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"kubernetes\" [] validServingFor=[kubernetes] issuer=\"<self>\" (2024-06-11 18:42:06 +0000 UTC to 2034-06-09 18:47:06 +0000 UTC (now=2024-06-18 20:52:34.488863005 +0000 UTC))" I0618 20:52:34.488907 1 tlsconfig.go:178] "Loaded client CA" index=11 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"front-proxy-ca\" [] validServingFor=[front-proxy-ca] issuer=\"<self>\" (2024-05-23 13:48:40 +0000 UTC to 2034-05-21 13:53:40 +0000 UTC (now=2024-06-18 20:52:34.488893022 +0000 UTC))" I0618 20:52:34.489320 1 tlsconfig.go:200] "Loaded serving cert" certName="serving-cert::/tmp/cert/apiserver.crt::/tmp/cert/apiserver.key" certDetail="\"localhost@1718743953\" [serving] validServingFor=[127.0.0.1,localhost,localhost] issuer=\"localhost-ca@1718743953\" (2024-06-18 19:52:33 +0000 UTC to 2025-06-18 19:52:33 +0000 UTC (now=2024-06-18 20:52:34.489306612 +0000 UTC))" I0618 20:52:34.489717 1 named_certificates.go:53] "Loaded SNI cert" index=0 certName="self-signed loopback" certDetail="\"apiserver-loopback-client@1718743954\" [serving] validServingFor=[apiserver-loopback-client] issuer=\"apiserver-loopback-client-ca@1718743953\" (2024-06-18 19:52:33 +0000 UTC to 2025-06-18 19:52:33 +0000 UTC (now=2024-06-18 20:52:34.489703978 +0000 UTC))" I0618 20:53:03.200349 1 handler.go:153] prometheus-metrics-adapter: GET "/healthz" satisfied by nonGoRestful I0618 20:53:03.200374 1 pathrecorder.go:241] prometheus-metrics-adapter: "/healthz" satisfied by exact match I0618 20:53:03.200563 1 httplog.go:132] "HTTP" verb="GET" URI="/healthz" latency="401.105µs" userAgent="kube-probe/1.28+" audit-ID="2cbe4861-95dc-46bc-a10b-f6f0d9431012" srcIP="172.28.153.122:44398" resp=200 I0618 20:53:03.200795 1 handler.go:153] prometheus-metrics-adapter: GET "/healthz" satisfied by nonGoRestful I0618 20:53:03.200815 1 pathrecorder.go:241] prometheus-metrics-adapter: "/healthz" satisfied by exact match I0618 20:53:03.200902 1 httplog.go:132] "HTTP" verb="GET" URI="/healthz" latency="202.339µs" userAgent="kube-probe/1.28+" audit-ID="18ac3f88-b9f3-4f7f-b867-9d43c509c2a0" srcIP="172.28.153.122:44396" resp=200 I0618 20:53:03.243908 1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1 I0618 20:53:03.243927 1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1 I0618 20:53:03.244185 1 httplog.go:132] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="978.503µs" userAgent="Go-http-client/2.0" audit-ID="44983ad4-fd2e-4f5a-9f30-4f81c4c1525a" srcIP="172.28.149.133:45582" resp=200 I0618 20:53:03.244369 1 httplog.go:132] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="1.088633ms" userAgent="Go-http-client/2.0" audit-ID="b38ca7e5-46b5-41c3-bbf5-e48e72a21321" srcIP="172.28.149.133:45582" resp=200 I0618 20:53:03.248643 1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1 I0618 20:53:03.248745 1 httplog.go:132] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="515.344µs" userAgent="Go-http-client/2.0" audit-ID="e38de6b7-a148-4b0f-88f3-761aab94c69d" srcIP="172.28.169.104:47184" resp=200 I0618 20:53:03.249284 1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1 I0618 20:53:03.249355 1 httplog.go:132] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="462.115µs" userAgent="Go-http-client/2.0" audit-ID="068f6f70-3a74-4a54-9651-b064a435ec96" srcIP="172.28.169.104:47184" resp=200 I0618 20:53:03.274046 1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1 I0618 20:53:03.274137 1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1 I0618 20:53:03.274196 1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1 I0618 20:53:03.274282 1 httplog.go:132] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="740.29µs" userAgent="Go-http-client/2.0" audit-ID="8d83f2c3-0fa4-43a5-a0c4-52e150205893" srcIP="172.28.149.133:45582" resp=200 I0618 20:53:03.274281 1 httplog.go:132] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="832.257µs" userAgent="Go-http-client/2.0" audit-ID="9f239e71-3711-45a2-a3f2-2b0553315006" srcIP="172.28.149.133:45582" resp=200 I0618 20:53:03.274166 1 httplog.go:132] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="625.659µs" userAgent="Go-http-client/2.0" audit-ID="d0b230fe-65de-47a6-b773-45cc98d84c7c" srcIP="172.28.149.133:45582" resp=200 I0618 20:53:03.274439 1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1 I0618 20:53:03.274532 1 httplog.go:132] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="1.013625ms" userAgent="Go-http-client/2.0" audit-ID="51788269-4dd1-4087-85c4-eb08bc1d90b0" srcIP="172.28.149.133:45582" resp=200 I0618 20:53:03.274138 1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1 I0618 20:53:03.274621 1 httplog.go:132] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="1.121974ms" userAgent="Go-http-client/2.0" audit-ID="9ba47584-2280-4749-b0ef-745c7c7ff1c3" srcIP="172.28.149.133:45582" resp=200 I0618 20:53:03.276251 1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1 I0618 20:53:03.276293 1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1 I0618 20:53:03.276403 1 httplog.go:132] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="739.248µs" userAgent="Go-http-client/2.0" audit-ID="a86d7d27-f62b-42b2-9e8f-b1e8056d51ce" srcIP="172.28.169.104:47184" resp=200 I0618 20:53:03.276408 1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1 I0618 20:53:03.276534 1 httplog.go:132] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="534.01µs" userAgent="Go-http-client/2.0" audit-ID="90b491ed-b0f8-4e7e-b832-aa53a3a83f3a" srcIP="172.28.169.104:47184" resp=200 I0618 20:53:03.276539 1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1 I0618 20:53:03.276427 1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1 I0618 20:53:03.276733 1 httplog.go:132] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="906.998µs" userAgent="Go-http-client/2.0" audit-ID="3b282115-d413-4bb6-9d60-efa951c6c077" srcIP="172.28.169.104:47184" resp=200 I0618 20:53:03.276608 1 httplog.go:132] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="810.341µs" userAgent="Go-http-client/2.0" audit-ID="fda2bcc7-1da8-494f-9c8f-1b73b3500518" srcIP="172.28.169.104:47184" resp=200 I0618 20:53:03.276620 1 httplog.go:132] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="740.651µs" userAgent="Go-http-client/2.0" audit-ID="d8651aad-cb38-47e8-9e97-71417c1d0ddd" srcIP="172.28.169.104:47184" resp=200 I0618 20:53:10.844979 1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1 I0618 20:53:10.845174 1 httplog.go:132] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="730.1µs" userAgent="Go-http-client/2.0" audit-ID="6ba8f934-9c87-4661-98e3-fe0c35cec209" srcIP="172.28.149.133:45582" resp=200 I0618 20:53:10.845210 1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1 I0618 20:53:10.845210 1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1 I0618 20:53:10.845216 1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1 I0618 20:53:10.845309 1 httplog.go:132] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="736.079µs" userAgent="Go-http-client/2.0" audit-ID="658648b9-0ad7-4d5c-8f7f-6a1cddda1198" srcIP="172.28.149.133:45582" resp=200 I0618 20:53:10.845309 1 httplog.go:132] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="737.778µs" userAgent="Go-http-client/2.0" audit-ID="97daae1f-eea2-4786-983a-2e09ce113a36" srcIP="172.28.149.133:45582" resp=200 I0618 20:53:10.845313 1 httplog.go:132] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="777.313µs" userAgent="Go-http-client/2.0" audit-ID="f3d9edd7-3595-4c17-8542-c76404e51eff" srcIP="172.28.149.133:45582" resp=200 I0618 20:53:10.845730 1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1 I0618 20:53:10.845828 1 httplog.go:132] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="1.274179ms" userAgent="Go-http-client/2.0" audit-ID="5f03c59e-34fa-4271-8ce4-13a39d3b078a" srcIP="172.28.149.133:45582" resp=200 I0618 20:53:13.208111 1 handler.go:153] prometheus-metrics-adapter: GET "/healthz" satisfied by nonGoRestful I0618 20:53:13.208202 1 pathrecorder.go:241] prometheus-metrics-adapter: "/healthz" satisfied by exact match I0618 20:53:13.208367 1 httplog.go:132] "HTTP" verb="GET" URI="/healthz" latency="426.914µs" userAgent="kube-probe/1.28+" audit-ID="1d6a3dff-3383-4bfc-8bc5-a33ac842d29d" srcIP="172.28.153.122:34046" resp=200 I0618 20:53:13.209188 1 handler.go:153] prometheus-metrics-adapter: GET "/healthz" satisfied by nonGoRestful I0618 20:53:13.209204 1 pathrecorder.go:241] prometheus-metrics-adapter: "/healthz" satisfied by exact match I0618 20:53:13.209305 1 httplog.go:132] "HTTP" verb="GET" URI="/healthz" latency="208.957µs" userAgent="kube-probe/1.28+" audit-ID="b8feeab8-6ce4-49af-ae55-5917506912c2" srcIP="172.28.153.122:34040" resp=200 I0618 20:53:13.813021 1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1 I0618 20:53:13.813101 1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1 I0618 20:53:13.813110 1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1 I0618 20:53:13.813167 1 httplog.go:132] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="783.736µs" userAgent="Go-http-client/2.0" audit-ID="2c4c450a-8994-4cb7-8ba3-f81ad1325f97" srcIP="172.28.169.104:47184" resp=200 I0618 20:53:13.813211 1 httplog.go:132] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="911.064µs" userAgent="Go-http-client/2.0" audit-ID="f9a6cc5a-751b-41c1-a91e-87a60d381c32" srcIP="172.28.169.104:47184" resp=200 I0618 20:53:13.813211 1 httplog.go:132] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="847.041µs" userAgent="Go-http-client/2.0" audit-ID="552e9408-714b-4889-94c8-1ab0898c4b5b" srcIP="172.28.169.104:47184" resp=200 I0618 20:53:13.813021 1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1 I0618 20:53:13.813327 1 httplog.go:132] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="944.958µs" userAgent="Go-http-client/2.0" audit-ID="aedc90f4-57f2-45f4-bd07-a5946d7b8120" srcIP="172.28.169.104:47184" resp=200 I0618 20:53:13.813021 1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1 I0618 20:53:13.813405 1 httplog.go:132] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="1.072883ms" userAgent="Go-http-client/2.0" audit-ID="09e70456-0e54-4182-9728-c32803491e5e" srcIP="172.28.169.104:47184" resp=200 I0618 20:53:23.201506 1 handler.go:153] prometheus-metrics-adapter: GET "/healthz" satisfied by nonGoRestful I0618 20:53:23.201537 1 pathrecorder.go:241] prometheus-metrics-adapter: "/healthz" satisfied by exact match I0618 20:53:23.201561 1 handler.go:153] prometheus-metrics-adapter: GET "/healthz" satisfied by nonGoRestful I0618 20:53:23.201583 1 pathrecorder.go:241] prometheus-metrics-adapter: "/healthz" satisfied by exact match I0618 20:53:23.201648 1 httplog.go:132] "HTTP" verb="GET" URI="/healthz" latency="291.14µs" userAgent="kube-probe/1.28+" audit-ID="11b83ba9-6534-400d-a982-962759c00e66" srcIP="172.28.153.122:54746" resp=200 I0618 20:53:23.201694 1 httplog.go:132] "HTTP" verb="GET" URI="/healthz" latency="213.265µs" userAgent="kube-probe/1.28+" audit-ID="2392fe0c-2868-4b45-ab04-a2450231d7ea" srcIP="172.28.153.122:54738" resp=200 I0618 20:53:28.895233 1 handler.go:133] prometheus-metrics-adapter: GET "/apis" satisfied by gorestful with webservice /apis I0618 20:53:28.895376 1 httplog.go:132] "HTTP" verb="GET" URI="/apis" latency="608.464µs" userAgent="" audit-ID="bfa3b3ef-66ee-4bde-abde-82b626085192" srcIP="172.28.149.133:42602" resp=304

Anything else we need to know?:

Environment:

• prometheus-adapter version:
  4.10.0
• prometheus version:
  prom/prometheus:v2.35.0
• Kubernetes version (use kubectl version):
  $k version
  Client Version: v1.30.1
  Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
  Server Version: v1.28.9-eks-036c24b
• Cloud provider or hardware configuration: aws
• Other info:

[dgrisonnet]

/kind support
/triage accepted

[pedropaulopereiraamarante]

any updates? ='(

[volkean]

If you install by helm upgrade --install prometheus-adapter prometheus-community/prometheus-adapter then it will show only custom metrics but if you enable external rules like helm upgrade --install prometheus-adapter prometheus-community/prometheus-adapter --set-json rules.external=[{}] then you will see external metrics as well.

[matthenry87]

If you install by helm upgrade --install prometheus-adapter prometheus-community/prometheus-adapter then it will show only custom metrics but if you enable external rules like helm upgrade --install prometheus-adapter prometheus-community/prometheus-adapter --set-json rules.external=[{}] then you will see external metrics as well.

We've always had that set in our values.yml, but ever since we upgraded our nodes to 1.30 it stops working.

[pedropaulopereiraamarante]

I will try, thank you!