Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for cgroupsv2 #791

Merged
merged 153 commits into from
Oct 20, 2023
Merged
Show file tree
Hide file tree
Changes from 64 commits
Commits
Show all changes
153 commits
Select commit Hold shift + click to select a range
434679d
[WIP] cgroups: implement cgroupsv2 support
globin Jul 2, 2021
6251cbe
[WIP] cgroups: default to cgroups v2 code temporarily
globin Aug 12, 2021
c6155a1
cgroups: remove debug logging
globin Aug 26, 2021
5b23179
cgroups: try with user service
globin Aug 26, 2021
66c22cd
cgroups: fix paths
globin Aug 26, 2021
9c6e3ff
cgroups: remove cgrulesd code in v2
globin Aug 26, 2021
133d5bf
cgroups: WIP abstraction for v1 & v2
globin Sep 26, 2021
3182bca
cgroups: fix permission hints
globin Sep 26, 2021
b4fe3dc
cgroups: WIP cleanup & fixes
globin Sep 27, 2021
c33516c
cgroups: cleanup some path handling
globin Sep 27, 2021
5f108ef
cgroups: fix issues from refactoring
globin Sep 27, 2021
c313298
cgroups: add cgroupsv2 bootstrap service
globin Oct 8, 2021
a67214b
cgroups: parameterise cgv2 fallback path
globin Oct 8, 2021
1d0e1c0
cgroups: OOM handling
globin Oct 12, 2021
e41913e
cgroups: remove unreferenced code
globin Oct 12, 2021
2307834
cgroups: cgroup.kill handling for >=5.14
globin Oct 12, 2021
4a354a9
cgroups: WIP IO and v2 CPU
globin Oct 12, 2021
1215096
cgroups: type-checking clean-up
globin Oct 12, 2021
89c5d64
cgroups: CI flake fixes?
globin Oct 12, 2021
f121081
cgroups: v2 CPU subsystem
globin Oct 14, 2021
6b1f75a
cgroups: cleanup and available mem node fix
globin Oct 14, 2021
34d2c8f
cgroups: initial v2 IO implementation
globin Oct 14, 2021
3206095
cgroups: fix v2 cgroup removal assertion
globin Oct 14, 2021
f92fea5
cgroups: update default branch name in links
globin Oct 14, 2021
26ab65a
cgroups: fix percpu usage output
globin Oct 14, 2021
4a4eacd
cgroups: fix resource handling
globin Oct 14, 2021
aa0b905
cgroups: implement oom/memlimit handling
globin Oct 20, 2021
1245dc0
cgroups: CI fixes
globin Oct 20, 2021
c815670
cgroups: implement non-fallback code path for v2
globin Oct 28, 2021
9dd3282
Merge remote-tracking branch 'origin/main' into cgroupsv2
globin Oct 28, 2021
78b39b0
Merge remote-tracking branch 'origin/main' into cgroupsv2
globin Oct 28, 2021
db9e6c0
cgroups: fix file/path handling edge cases
globin Nov 2, 2021
f9481ed
Merge remote-tracking branch 'origin/main' into cgroupsv2
globin Nov 24, 2021
ffd05e5
cgroups: use pystemd with cgroupsv2
globin Nov 25, 2021
bab92a5
cgroups: only allow execution in proper cgroup
globin Dec 2, 2021
8ea6777
cgroups: add some->total pressure metrics for cgroups v2
globin Dec 4, 2021
3469607
cgroups: partially revert cgroup-availability checking
globin Dec 4, 2021
5db0ec8
cgroups: cleanup
globin Dec 4, 2021
f7b369c
Merge remote-tracking branch 'origin/main' into cgroupsv2
globin Jan 9, 2022
5b4f309
move benchexec's process group to scope not runexec
globin Jan 11, 2022
21fdde2
add pressure metrics to test whitelist
globin Jan 11, 2022
a5a8a00
Merge remote-tracking branch 'origin/main' into cgroupsv2
globin Jan 11, 2022
c3d7c1c
cgroups v2: remove fallback logic
globin Jan 11, 2022
4d32b32
util.get_pgrp_pids: fix race condition
globin Jan 12, 2022
9728457
benchmark-cgroup2.service: remove
globin Jan 12, 2022
11d874a
Merge remote-tracking branch 'origin/main' into cgroupsv2
globin Feb 26, 2022
b9d2c63
tests: do not skip if executable is on PATH
globin Feb 27, 2022
567480b
tests: fix cgroup v2 tests
globin Feb 27, 2022
2212ff6
tests: reformat
globin Feb 27, 2022
4eec9e0
gitlab-ci: install pystemd
globin Feb 27, 2022
6341f71
gitlab-ci: actually install pystemd
globin Feb 27, 2022
ab65a52
gitlab-ci: install libsystemd and python headers instead
globin Feb 28, 2022
3576599
tests: fix typo
globin Feb 28, 2022
758ceec
gitlab-ci: fix pytype
globin Feb 28, 2022
d1691bd
gitlab-ci: really fix pytype?
globin Feb 28, 2022
2a34e76
gitlab-ci: really fix pytype?!
globin Feb 28, 2022
f4c3f0e
cgroups: make sure paths are strings before escaping
globin Mar 7, 2022
daad5e4
check_cgroups: fix for cgv2
globin Mar 7, 2022
3e27f1d
test_runexecutor: use existing metric for cgv2
globin Mar 7, 2022
d70a66b
util.get_prgp_pids: fix when binaries with spaces are running
globin Mar 7, 2022
a6bf3d2
util.get_prgp_pids: fix typo
globin Mar 7, 2022
c42b142
util.get_prgp_pids: rerun black
globin Mar 7, 2022
7386ec2
Merge remote-tracking branch 'origin/main' into cgroupsv2
globin Mar 7, 2022
3b776d5
test_runexecutor: fix typo
globin Mar 7, 2022
0ae1aad
Merge branch 'main' into cgroupsv2
PhilippWendler Jul 22, 2022
4f4aff8
Port a921d6be to this branch
PhilippWendler Jul 22, 2022
568c614
Revert change to test parameter that does not belong to cgroupsv2 branch
PhilippWendler Jul 27, 2022
3f3bcd0
Simplify creation of random token and silence flake8 warning
PhilippWendler Jul 22, 2022
28a2aa3
Fix containerexec: it should not use cgroups
PhilippWendler Jul 22, 2022
3829226
Fix bug in cgroups handling: has_tasks() always returned False
PhilippWendler Jul 26, 2022
ffe3913
Fix handling of child cgroups
PhilippWendler Jul 26, 2022
9cc8deb
Add method from Cgroups classes also to abstract base class
PhilippWendler Jul 27, 2022
885e12c
Remove special casing for cgroup versions from RunExecutor
PhilippWendler Jul 27, 2022
86ff57b
Make some methods on CgroupsV2 easier to use
PhilippWendler Jul 27, 2022
9729ef5
Rewrite main logic for initializing cgroups
PhilippWendler Jul 27, 2022
a2eabf1
Make check_cgroups work properly for cgroups v2
PhilippWendler Jul 27, 2022
da642c9
Refactoring: Move fields and methods from instance to class for Cgroups
PhilippWendler Jul 27, 2022
3e47e4b
Refactoring: simplify creation of Cgroups instances
PhilippWendler Jul 27, 2022
873759c
Remove restriction to well-known controllers for cgroups v2
PhilippWendler Jul 27, 2022
84535f8
Refactor CgroupsV2.create_fresh_child_cgroup()
PhilippWendler Jul 27, 2022
ddf5ad0
Do not throw BenchExecException from cgroups module
PhilippWendler Jul 27, 2022
dc8adf0
More robust parsing of io.stat on cgroups v2
PhilippWendler Jul 27, 2022
f350ed6
Fix output of pressure stall time with cgroups v2
PhilippWendler Jul 27, 2022
9fcf85c
First step at better error handling for cgroups v2
PhilippWendler Aug 30, 2022
38fe93b
Merge commit '5f48889dfa52d5ac130bdd38e4d9c8ff6940cbfb' into cgroupsv2
PhilippWendler Sep 15, 2022
5d44299
format
PhilippWendler Sep 15, 2022
c4e447c
Copy over protection for #840 to cgroupsv2 branch
PhilippWendler Feb 27, 2023
d150d47
Merge commit 'b3462dc69049a863c1309b39fb65883eba335c33' into cgroupsv2
PhilippWendler Feb 27, 2023
b0f5eb2
Merge commit '43989c1ef97f42d773b9e99796eef5a2b8ca7742' into cgroupsv2
PhilippWendler Feb 27, 2023
efc4f19
Merge formatting commit '894f9524' into cgroupsv2 and reformat
PhilippWendler Feb 27, 2023
2efefe1
Merge branch 'main' into cgroupsv2
PhilippWendler Mar 17, 2023
b5e8d62
Remove ZFS from list of filesystems that do not support overlayfs
PhilippWendler Mar 17, 2023
a302f70
Fix method call from c4e447ce
PhilippWendler Mar 17, 2023
921929b
Remove inconsistent pathlib usage from cgroupsv1 code
PhilippWendler Mar 17, 2023
8a7f613
Fix broken merge of 314b32b1 intro branch cgroupsv2
PhilippWendler Jan 30, 2023
a869c00
Remove pystemd as "build dependency" and make it an optional dependency
PhilippWendler Mar 17, 2023
7f8f5b4
remove test adjustment that is now unnecessary
PhilippWendler Mar 17, 2023
92ad434
Silence pytype error about potentially undefined name
PhilippWendler Mar 17, 2023
21a4a35
Merge branch 'main' into cgroupsv2
PhilippWendler Mar 17, 2023
5dd0006
Merge branch 'main' into cgroupsv2
PhilippWendler Mar 17, 2023
1596a6e
Remove forgotten code from code move back to oomhandler.py in aa0b905f
PhilippWendler Mar 17, 2023
51c23c4
Refactoring: Move code around
PhilippWendler Mar 17, 2023
3dcb47c
Remove more pathlib usage from cgroupsv1 code
PhilippWendler Mar 17, 2023
81bfdbf
Rename some methods to keep the code more similar with existing code
PhilippWendler Mar 30, 2023
4b4f939
Make comments and abstract methods for Cgroups class more consistent
PhilippWendler Mar 30, 2023
040fcb6
Move print_decimal to benchexec.util
PhilippWendler Mar 30, 2023
8ab8111
Fix missing log message of benchexec.check_cgroups in cgroupsv2
PhilippWendler Mar 30, 2023
39808b6
Remove redundant Cgroups.initialize() call
PhilippWendler Mar 30, 2023
30e8417
Merge branch 'main' into cgroupsv2
PhilippWendler Mar 30, 2023
e77404f
Move methods around to put them in better order
PhilippWendler Mar 30, 2023
f787fae
Simplify kill_all_tasks_in_cgroups
PhilippWendler Mar 30, 2023
cd8dad9
Simplify handling of process termination on cgroupsv2
PhilippWendler Apr 3, 2023
e324181
Fix bug for cgroupsv2: child cgroups were not deleted
PhilippWendler Apr 3, 2023
255ac51
Merge branch 'main' into cgroupsv2
PhilippWendler Apr 3, 2023
6cb8faf
Guard against child cgroups with unexpected permissions for cgroupsv2
PhilippWendler Apr 3, 2023
ff28210
Port and enable test with frozen processes for cgroupsv2
PhilippWendler Apr 3, 2023
cbf6543
Remove some unimportant logging that clutters the output
PhilippWendler Apr 3, 2023
06ae55b
Implement TODO: improve interface of Cgroups.has_tasks()
PhilippWendler Apr 3, 2023
419cae3
Avoid crash in read_memory_limit() if limit is "max"
PhilippWendler Apr 3, 2023
7aec1e2
Check preexisting memory limit also on cgroupsv2
PhilippWendler Apr 3, 2023
51f39fa
Refactoring: refactor out common logic
PhilippWendler Apr 3, 2023
2271d1c
Always read pressure stall information from cgroupsv2
PhilippWendler Apr 3, 2023
9f1029c
Refactoring: Implement reading of pressure information only once
PhilippWendler Apr 3, 2023
327cd01
Rename and document the pressure values that runexec returns
PhilippWendler Apr 3, 2023
2eb1f91
Fix missing units for pressure information in XML results
PhilippWendler Apr 3, 2023
d3d1d74
Implement reading of memory usage for cgroupsv2
PhilippWendler Apr 3, 2023
88c337e
Slight refactorings for reading cgroup values
PhilippWendler Apr 3, 2023
1c98bc4
Fix type error found by pytype
PhilippWendler Apr 3, 2023
590ac51
Fix warning about swapaccount being always shown for cgroupsv2
PhilippWendler Apr 3, 2023
ae71489
Detect OOM kills only in our cgroup, not in child cgroups (for cgroup…
PhilippWendler Apr 4, 2023
6938ef8
Fix OOM handling on cgroupsv2
PhilippWendler Apr 5, 2023
f2839b7
Improve OOM handling for cgroupsv2
PhilippWendler Apr 5, 2023
e428fa2
fix failure in case key does not exist
PhilippWendler Apr 5, 2023
29d7cfa
fix logic bug
PhilippWendler Apr 5, 2023
cb7403e
Merge branch 'main' into cgroupsv2
PhilippWendler Jul 12, 2023
a5655f4
Put our systemd scope into a separate slice
PhilippWendler Apr 5, 2023
2f3aa04
Require that we are the only process in our cgroup for cgroupsv2
PhilippWendler Apr 6, 2023
07c69b9
Merge branch 'main' into cgroupsv2
PhilippWendler Aug 24, 2023
14d8210
Use cgroup namespace for runs with cgroupsv2
PhilippWendler Aug 24, 2023
f63bb3d
Refactoring: Move code to separate method to allow reuse
PhilippWendler Aug 24, 2023
a6d7d36
Add another nesting level of cgroups in case of cgroup namespaces
PhilippWendler Aug 24, 2023
f0ad1df
Add --cgroup-access to containerexec for cgroupsv2
PhilippWendler Aug 24, 2023
0cc4975
add some type annotations and checks to help pytype
PhilippWendler Aug 24, 2023
6cbd9fc
Better error message for problem of missing cpuset delegation
PhilippWendler Aug 24, 2023
2a0eec2
code style
PhilippWendler Sep 5, 2023
d17198e
Update Debian package config for cgroupsv2
PhilippWendler Oct 19, 2023
40d3986
Update installation instructions with cgroups v2
PhilippWendler Oct 19, 2023
fa8a85d
Handle failure case of r/o cgroupfs for cgroupsv2
PhilippWendler Sep 6, 2023
93f22e7
more documentation for cgroups v2
PhilippWendler Oct 19, 2023
2d08b1d
Merge branch 'main' into cgroupsv2
PhilippWendler Oct 19, 2023
d97af63
polish documentation for cgroups v2
PhilippWendler Oct 20, 2023
3b2a933
Let check_cgroups produce better error message for missing CPUSET
PhilippWendler Oct 20, 2023
f8348a7
Silence warning about missing CPUSET if not required
PhilippWendler Oct 20, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .gitlab-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ stages:
# Give $PRIMARY_USER permission to create cgroups
- test/for_each_of_my_cgroups.sh chgrp $PRIMARY_USER
- test/for_each_of_my_cgroups.sh chmod g+w $PRIMARY_USER
- apt install -y libsystemd-dev python-dev
PhilippWendler marked this conversation as resolved.
Show resolved Hide resolved
# Install BenchExec
- sudo -u $PRIMARY_USER pip install --user .
# Start lxcfs
Expand Down Expand Up @@ -114,8 +115,10 @@ pytype:
stage: test
image: python:3.7
before_script:
- apt update
- apt install -y libsystemd-dev python-dev
PhilippWendler marked this conversation as resolved.
Show resolved Hide resolved
# version due to https://github.com/google/pytype/issues/1130
- pip install coloredlogs pytype==2022.2.8
- pip install pystemd coloredlogs pytype==2022.2.8
script:
- pytype -k
cache:
Expand Down
Loading