Adjust cadvisor container settings

[drmatthews]

According to the cadvisor docs when running the container on a RedHat 7 host it should be run as privileged. The mounted volumes have been adjusted to include /sys/fs/cgroup/cpu,cpuacct:/sys/fs/cgroup/cpuacct,cpu. However, the container still shows as unhealthy when running the molecule tests.

On the monitoring host (mirsg-linux) the deployed container is also showing as unhealthy but the logs have lots of entries like:

W0111 18:01:59.875044       1 container.go:549] Failed to update stats for container "/docker/fbbd1164db6eaf5e18d046532fdbda77ba4ea54a4e676f46eab939370804d4f1": failed to parse memory.usage_in_bytes - read /sys/fs/cgroup/memory/docker/fbbd1164db6eaf5e18d046532fdbda77ba4ea54a4e676f46eab939370804d4f1/memory.usage_in_bytes: no such device, continuing to push stats

The above error is generated because the container it refers to no longer exists (fbbd1164db6e) so collecting metrics fails; the cadvisor container however continues to run but is indicating unhealthy. Recreating the cadvisor container (stopping and removing manually and then redeploying with Ansible) results in an unhealthy status but with no errors in the logs.

[p-j-smith]

looks good

just to check, even with these changes the container is still unhealthy? Should we look for an alternative to cadvisor?

[drmatthews]

just to check, even with these changes the container is still unhealthy?

Apologies for being unclear. Yes, still unhealthy in the tests. The actual deployed container is also still unhealthy but I managed to get rid of the error messages in the logs.

Should we look for an alternative to cadvisor?

I'll create an issue. Right now it isn't really doing anything other than collecting metrics from the various prometheus related containers (which we're not alerting on).