Cadvisor Metrics Label contain empty Strings as values (container="", image="",...)

[Feederhigh5]

What Happened?

I want to analyze the resource consumption (CPU, Network, Memory, I/O) on container level.
As far as I know, these metrics are generated by cadvisor and exposed at https://kubernetes.default.svc/api/v1/nodes/minikube/proxy/metrics/cadvisor.

I check them through the following command: kubectl get --raw /api/v1/nodes/minikube/proxy/metrics/cadvisor

My problem is, that the necessary container label (among others like image and name) are empty strings. That prevents me from analyzing the resource consumption per container.
For example:

container_network_receive_bytes_total{container="",id="/",image="",interface="bridge",name="",namespace="",pod=""} 2.90072711e+08 1687270944453

container_memory_working_set_bytes{container="",id="/kubepods/pod49686167-c619-4f80-bfd7-5bacc76e2e47",image="",name="",namespace="logging",pod="kibana-kibana-749869bc5c-4dgmt"} 4.0378368e+08 1687269135477

How do I configure cadvisor (or check if cadvisor is correctly configured)?
Do you have a suggestion how I could get these metrics?

I would highly appreciate help! If you need any more info, let me know.
Thank you :)

Attach the log file

minikube_log.txt

Operating System

Windows

Driver

Docker

[mdamour1976]

Seeing the same issue. When using --kubernetes-version=v1.23.17 it works, so something changed between then and v1.24.x (and present).

[davidmirza408]

I am experiencing the same problem while running version 1.28 on darwin with a minikube setup.

container_threads_max{container="",id="/kubepods/burstable/pod3eadeaa9-ea2b-4306-8d36-4362544e97f1/89f9ece7f73db56c1703176ea98328158427b7c771db42923261b48827d0fcb4",image="",name="",namespace="",pod=""} 0 1701391500461
container_threads_max{container="",id="/kubepods/burstable/pod3eadeaa9-ea2b-4306-8d36-4362544e97f1/ea931fff7d2b5e0bf4d1723ec4ca5b9c4e21ee6d5b32115838830bbc265fec96",image="",name="",namespace="",pod=""} 0 1701391497962

[harshitrjpt]

Try containerd as the container runtime when starting minikube.
I believe K8 1.23+ or 1.24+ (not sure which exactly) but containerd has been the default container runtime. Maybe that is related somehow here.
In my local i was facing the same issue (default is docker runtime), so i switched to containerd

minikube start --container-runtime=containerd

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.