Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clean up unused checkouts on dev-desktops #389

Merged
merged 3 commits into from
Mar 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions ansible/roles/dev-desktop/tasks/cleanup.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
---
- name: Copy cleanup script
template:
src: clean-unused-checkouts.sh
dest: /etc/cron.cleanup_disk_space
owner: root
group: root
mode: 0744

- name: Set up the cleanup cron job
template:
src: cron_cleanup_disk_space.j2
dest: /etc/cron.d/cleanup_disk_space
# if the cron job is running right now, keep retrying until it finishes
register: cleanup_cron_result
until: cleanup_cron_result is not failed
retries: 10
delay: 5
1 change: 1 addition & 0 deletions ansible/roles/dev-desktop/tasks/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
- include_tasks: usermod.yml
with_items: "{{ vars_user_config }}"
when: vars_user_config is not none and vars_user_config | length > 0
- include_tasks: cleanup.yml
- include_tasks: team_login.yml
- include_tasks: github.yml
- include_tasks: motd.yml
Expand Down
2 changes: 1 addition & 1 deletion ansible/roles/dev-desktop/tasks/team_login.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@

- name: Set up the team login cron job
template:
src: crontab_append
src: cron_team_login.j2
dest: /etc/cron.d/team_login
# if the cron job is running right now, keep retrying until it finishes
register: task_result
Expand Down
74 changes: 74 additions & 0 deletions ansible/roles/dev-desktop/templates/clean-unused-checkouts.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
#!/usr/bin/env bash

#
# {{ ansible_managed }}
#

# Clean up unused checkouts
#
# This script is used to find old checkouts that are no longer in use. Given the
# size of the build directory, regularly cleaning them up can save significant
# amounts of disk space.

# Enable strict mode for Bash
# http://redsymbol.net/articles/unofficial-bash-strict-mode/
set -euo pipefail
IFS=$'\n\t'

# Print directories and their size instead of deleting them
dry_run=false

# Default to search for checkouts older than 60 days
time="60"

while [[ $# -gt 0 ]]; do
case $1 in
--dry-run)
dry_run=true
shift # past argument
;;
-t|--time)
time="${2}"
shift # past argument
shift # past value
;;
-*)
echo "Unknown option $1"
exit 1
;;
esac
done

# Find all build or target directories created by users
#
# This command combines (`-o`) two different conditions to find all build and
# target directories that users have created. Within each home directory, we
# recursively look for directories that either have a file named `x.py` and a
# directory named `build`, or a file named `Cargo.toml` and a directory named
# `target`.
all_cache_directories=$(find /home -type d \( -name build -execdir test -f "x.py" \; -o -name target -execdir test -f "Cargo.toml" \; \) -print | sort | uniq)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You may also want to sweep build-rust-analyzer too, which is a secondary build dir recommended for r-a workflows: https://rustc-dev-guide.rust-lang.org/building/suggested.html#configuring-rust-analyzer-for-rustc

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll make a note of this in #390, which tracks rewriting the script in Rust. I fear adding more conditions to the find command will get unmanageable quickly. 🙈


# For each checkout, we want to determine if the user has been working on it
# within the `$time` number of days.
unused_cache_directories=$(for directory in $all_cache_directories; do
project=$(dirname "${directory}")

# Find all directories with files that have been modified less than $time days ago
modified=$(find "${project}" -type f -mtime -"${time}" -printf '%h\n' | xargs -r dirname | sort | uniq)

# If no files have been modified in the last 90 days, then the project is
# considered old.
if [[ -z "${modified}" ]]; then
echo "${directory}"
fi
done)

# Delete the build directories in the unused checkouts
for directory in $unused_cache_directories; do
if [[ "${dry_run}" == true ]]; then
du -sh "${directory}"
else
echo "Deleting ${directory}"
rm -rf "${directory}"
fi
done
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
0 0 * * * root /etc/cron.cleanup_disk_space
Loading