Kepler didn't export pod metrics

[loicmathieu]

Hi,
We just deployed Kepler in our Kubernetes cluster to test it and we saw almost no metrics exported.
The only metrics that I can see are these ones :

And there is some error in the export logs:

I1116 10:27:08.392204       1 exporter.go:81] Kepler running on version:
I1116 10:27:08.392266       1 config.go:89] using gCgroup ID in the BPF program: true
I1116 10:27:08.392307       1 config.go:90] kernel version: 5.1
I1116 10:27:08.392347       1 exporter.go:106] Initializing the Model Server
I1116 10:27:08.635605       1 slice_handler.go:189] Not able to find any valid .scope file in /sys/fs/cgroup/cpu, this likely cause all cgroup metrics to be 0
I1116 10:27:08.729001       1 exporter.go:114] Initializing the GPU collector
perf_event_open: No such file or directory
I1116 10:27:10.350973       1 bcc_attacher.go:90] failed to attach perf event cpu_instr: failed to open bpf perf event: no such file or directory
perf_event_open: No such file or directory
I1116 10:27:10.351043       1 bcc_attacher.go:90] failed to attach perf event cache_miss: failed to open bpf perf event: no such file or directory
perf_event_open: No such file or directory
I1116 10:27:10.351114       1 bcc_attacher.go:90] failed to attach perf event cpu_cycles: failed to open bpf perf event: no such file or directory
I1116 10:27:10.351129       1 bcc_attacher.go:131] Successfully load eBPF module with option: [-DNUM_CPUS=8 -DCPU_FREQ]
I1116 10:27:10.376561       1 exporter.go:152] Started Kepler in 1.984403654s

This is a GKE cluster, does Kelpler already been tested in GKE ?

[rootfs]

@loicmathieu thank for getting this issue report. Can you check the version of the kepler pod? There are a couple of fixes to solve the VM based deployment these days.

kubectl get pod -n kepler -o yaml |grep imageID

[rootfs]

thank you for the update, @loicmathieu !

Do you mind setting v=5 to increase verbosity and getting the log?
https://github.com/sustainable-computing-io/kepler/blob/main/manifests/kubernetes/base/deployment.yaml#L76

You can find me at slack (invite sent to your email) and we can sync over there

[sunya-ch]

We still need a couple of PRs to make the power model work on the VM.
At least this #397 need to be merged to make the Kepler properly detect the cgroup metrics.
According to the log you gave above, there is no cgroup metric reported yet.

I1116 10:27:08.635605       1 slice_handler.go:189] Not able to find any valid .scope file in /sys/fs/cgroup/cpu, this likely cause all cgroup metrics to be 0
I1116 10:27:08.729001       1 exporter.go:114] Initializing the GPU collector

[loicmathieu]

Here is a detail log.

I1116 15:02:01.010517       1 metric_collector.go:132] energy from pod (0 processes): name: kepler-exporter-9g9hg namespace: kepler
        cgrouppid: 0 pid: [] comm:
        ePkg (mJ): 5326 (314234) (eCore: 5326 (314234) eDram: 5318 (313762) eUncore: 0 (0)) eGPU (mJ): 0 (0) eOther (mJ): 0 (0)
        eDyn (mJ): 0 (0)
        avgFreq: 0.00
        CPUTime:  0 (0)
        counters: map[cache_miss:0 (0) cpu_cycles:0 (0) cpu_instr:0 (0)]
        cgroupfs: map[]
        kubelets: map[container_cpu_usage_seconds_total:0 (3) container_memory_working_set_bytes:212992 (109293568)]

[loicmathieu]

@sunya-ch as soon as #397 will be merged and a new container been published I can try it.
I can send you the list of files in the /sys/fs/cgroup/cpu in GKE if you need it.

[loicmathieu]

Tested with a custom build with #397 and it works fine.