-
Notifications
You must be signed in to change notification settings - Fork 25
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Auto-generated from v0.1.0-38-g246e1f7 by 'update-gh-pages.sh'
- Loading branch information
Github
committed
Sep 29, 2023
1 parent
611b59a
commit 6785ffd
Showing
29 changed files
with
713 additions
and
37 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| # Sphinx build info version 1 | ||
| # This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. | ||
| config: 096e5cb6a6b51a425ab1cbc90a0b8e52 | ||
| config: 075dee4d7e63f1e6bbb9b28fa166bb14 | ||
| tags: 645f666f9bcd5a90fca523b33c5a78b7 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| # Memory plugins | ||
|
|
||
| ```{toctree} | ||
| --- | ||
| maxdepth: 2 | ||
| caption: Contents | ||
| --- | ||
| memory-qos.md | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,155 @@ | ||
| # Memory QoS NRI plugin | ||
|
|
||
| This NRI plugin adds two methods for controlling cgroups v2 `memory.*` | ||
| parameters: memory QoS classes and direct memory annotations. | ||
|
|
||
| ## Workload configuration | ||
|
|
||
| There are two configuration methods: | ||
| 1. Memory QoS classes: memory parameters are calculated in the same | ||
| way for all workloads that belong to the same class. | ||
| 2. Direct workload-specific memory parameters. | ||
|
|
||
| Memory QoS class of a pod or a container is defined using annotations | ||
| in pod yaml: | ||
|
|
||
| ```yaml | ||
| annotations: | ||
| # Set the default memory QoS class for all containers in this pod. | ||
| class.memory-qos.nri.io: silver | ||
|
|
||
| # Override the default class for the c0 container. | ||
| class.memory-qos.nri.io/c0: bronze | ||
|
|
||
| # Remove the default class from the c1 container. | ||
| class.memory-qos.nri.io/c1: "" | ||
| ``` | ||
|
|
||
| Cgroups v2 memory parameters are given pod annotations. Following | ||
| example affects `memory.swap.max`, `memory.high` and | ||
| `memory.oom.group`: | ||
|
|
||
| ```yaml | ||
| annotations: | ||
| # Never swap memory of the noswap container in this pod. | ||
| memory.swap.max.memory-qos.nri.io/noswap: "0" | ||
| memory.high.memory-qos.nri.io/noswap: max | ||
|
|
||
| # For all containers: if a process gets OOM killed, | ||
| # do not group-kill the whole cgroup. | ||
| memory.oom.group.memory-qos.nri.io: "0" | ||
| ``` | ||
|
|
||
| ## Plugin configuration | ||
|
|
||
| ### Classes | ||
|
|
||
| Plugin configuration lists memory QoS classes and their parameters | ||
| that affect calculating actual memory parameters. | ||
|
|
||
| `classes:` is followed by list of maps with following keys and values: | ||
| - `name` (string): name of the memory QoS class, matches | ||
| `class.memory-qos.nri.io` annotation values. | ||
| - `swaplimitratio` (from 0.0 to 1.0): minimum ratio of container's | ||
| memory on swap and resources.limits.memory when container's memory | ||
| consumption reaches the limit. Adjusts `memory.high` watermark to | ||
| `resources.limits.memory * (1.0 - swaplimitratio)`. | ||
|
|
||
| ### Unified annotations | ||
|
|
||
| `unifiedannotations:` (list of strings): OCI Linux unified fields | ||
| (cgroups v2 file names) whose values are allowed to be set using | ||
| direct annotations. If annotations define these values, they override | ||
| values implied by container's memory QoS class. | ||
|
|
||
| ### Example | ||
|
|
||
| ```yaml | ||
| classes: | ||
| - name: bronze | ||
| swaplimitratio: 0.5 | ||
| - name: silver | ||
| swaplimitratio: 0.2 | ||
| unifiedannotations: | ||
| - memory.swap.max | ||
| - memory.high | ||
| ``` | ||
|
|
||
| This configuration defines the following. | ||
|
|
||
| - If a container belogs to the memory QoS class `bronze` has allocated | ||
| half of the memory of its `resources.limits.memory`, next | ||
| allocations will cause kernel to swap out corresponding amount of | ||
| container's memory. In other words, when container's memory usage is | ||
| close to the limit, at most half of its data is stored in RAM. | ||
| - Containers in `silver` class are allowed to keep up to 80 % of their | ||
| data in RAM when reaching memory limit. | ||
| - Memory annotations are allowed to modify `memory.swap.max` and | ||
| `memory.high` values directly but, for instance, modifying | ||
| `memory.oom.group` is not enabled by this configuration. | ||
|
|
||
| ## Developer's guide | ||
|
|
||
| ### Prerequisites | ||
|
|
||
| - Containerd v1.7+ | ||
| - Enable NRI in /etc/containerd/config.toml: | ||
| ``` | ||
| [plugins."io.containerd.nri.v1.nri"] | ||
| disable = false | ||
| disable_connections = false | ||
| plugin_config_path = "/etc/nri/conf.d" | ||
| plugin_path = "/opt/nri/plugins" | ||
| plugin_registration_timeout = "5s" | ||
| plugin_request_timeout = "2s" | ||
| socket_path = "/var/run/nri/nri.sock" | ||
| ``` | ||
|
|
||
| ### Build | ||
|
|
||
| ```bash | ||
| cd cmd/plugins/memory-qos && go build . | ||
| ``` | ||
|
|
||
| ### Run | ||
|
|
||
| ```bash | ||
| cmd/plugins/memory-qos/memory-qos -config sample-configs/nri-memory-qos.yaml -idx 40 -vv | ||
| ``` | ||
|
|
||
| ### Manual test | ||
|
|
||
| ```bash | ||
| kubectl create -f test/e2e/files/nri-memory-qos-test-pod.yaml | ||
| ``` | ||
|
|
||
| See swap status of dd processes, each allocating the same amount of | ||
| memory: | ||
|
|
||
| ```bash | ||
| for pid in $(pidof dd); do | ||
| grep VmSwap /proc/$pid/status | ||
| done | ||
| ``` | ||
|
|
||
| ### Debug | ||
|
|
||
| ```bash | ||
| go install github.com/go-delve/delve/cmd/dlv@latest | ||
| dlv exec cmd/plugins/memory-qos/memory-qos -- -config sample-configs/nri-memory-qos.yaml -idx 40 | ||
| (dlv) break plugin.CreateContainer | ||
| (dlv) continue | ||
| ``` | ||
|
|
||
| ### Deploy | ||
|
|
||
| Build an image, import it on the node, and deploy the plugin by | ||
| running the following in `nri-plugins`: | ||
|
|
||
| ```bash | ||
| rm -rf build | ||
| make clean | ||
| make PLUGINS=nri-memory-qos IMAGE_VERSION=devel images | ||
| ctr -n k8s.io images import build/images/nri-memory-qos-image-*.tar | ||
| kubectl create -f build/images/nri-memory-qos-deployment.yaml | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.