Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cadvisor: Fix pod detection for containerd runtime on k8s #189

Merged
merged 8 commits into from
Mar 24, 2021

Conversation

pingleig
Copy link
Member

@pingleig pingleig commented Mar 22, 2021

Description of the issue

Fix #188 k8s cluster using containerd runtime was not able to get pod metrics because

  • contained socket is not mounted
  • pod detection logic relies on pause container with name 'POD`, which is only the case for docker

Description of changes

  • Define a new constant TypeInfraContainer and remove most usage on POD container name.
  • update all the manifest to mount containerd
  • deal with kubernetes>docker>container_id not sure if it is used by any place (it comes from k8s api server though, not cadvisor), just use containerd in json path when it is containerd
  • container file system metric like container_filesystem_utilization is missing in structured log [k8s][containerd] container file system metrics is not supported by cadvisor for contianerd #192, it's known issue in cadviisor.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

  • containerd cluster using kops w/ k8s version 1.20.4
  • docker cluster using kops/eks

With proper sock mounting, some path still got empty label and we should
not generate warning error message.
@@ -140,6 +140,9 @@ spec:
- name: varlibdocker
mountPath: /var/lib/docker
readOnly: true
- name: containerdsock
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just want to mention that the yaml files should also be updated in another repo: https://github.com/aws-samples/amazon-cloudwatch-container-insights

@pingleig pingleig added this to the v1.247347.6 milestone Mar 23, 2021
It's a bit annoying ... but it helps when user turn on debug log
* Skip diskio because we only extract node disk io
Existing targets build binary for all platforms, but for debugging
container related problems, most time we only need some binaries for
amd64.
@pingleig pingleig changed the title [WIP] cadvisor: Fix pod detection for containerd runtime on k8s cadvisor: Fix pod detection for containerd runtime on k8s Mar 24, 2021
@pingleig pingleig merged commit 804ba5b into aws:master Mar 24, 2021
pingleig added a commit to pingleig/amazon-cloudwatch-container-insights that referenced this pull request Apr 22, 2021
pingleig added a commit to aws-samples/amazon-cloudwatch-container-insights that referenced this pull request Apr 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[k8s] Pod metrics is gone when using containerd as runtime
2 participants