Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Automated workspace pruning/cleaning #35

Merged
merged 32 commits into from
Jul 15, 2024
Merged

Automated workspace pruning/cleaning #35

merged 32 commits into from
Jul 15, 2024

Conversation

wangpatrick57
Copy link
Member

@wangpatrick57 wangpatrick57 commented Jul 8, 2024 •
edited
Loading

Summary: A command to find and remove unneeded files from the workspace directory.

Demo:
Passing 25 unit tests with many edge cases which create files/directories, call clean_workspace(), and then verify its contents.
Screenshot 2024-07-07 at 17 05 07

Details

  • "Aggressive" mode removes all task_runs/*/ directories that are not directly pointed to by a symlink in symlinks/.
  • "Safe" mode also keeps task_runs/*/ directories which are indirectly pointed to by a symlink. This can happen if a symlink points to a task_runs/*/ directory which has a symlink in it that points to another task_runs/*/ directory.
  • I chose to write so many unit tests because this operation must be bug-free.

@wangpatrick57
basic os.walk
642a77f
@wangpatrick57
config group -> manage group
6c93ae4
@wangpatrick57
Merge branch 'main' into clean-workspace
699d1f8
@wangpatrick57
renamed unit test folders
ff122d0
@wangpatrick57
create_structure function
20aa6ed
@wangpatrick57
del scratchspace
7dbf90d
@wangpatrick57
wrote test_structure_helpers
5addc5f
@wangpatrick57
wrote basic clean_workspace
8538f66
@wangpatrick57
fixed create_structure to do symlinks from root_path
1a6b204
@wangpatrick57
comments for more tests for clean
768cdb9
@wangpatrick57
wrote empty tests
2e766f8
@wangpatrick57
refactored parent_dir and dir_basename functions
155def0
@wangpatrick57
merged with main
366e288
@wangpatrick57
fixed linking to dirs
49d5e64
@wangpatrick57
fixed file in dir in runs
702afa1
@wangpatrick57
link to link crashing now
b42a095
@wangpatrick57
added test for aggressive mode
e0b4743
@wangpatrick57
fixed verify structure to be fine with broken symlinks
35faba2
@wangpatrick57
tests for looping infinitely
059ffd2
@wangpatrick57
merge
8106f02
@wangpatrick57
fixed infinite loop problem
64a555b
@wangpatrick57
added some tests
2592db5
@wangpatrick57
added more tests
2d41c9a
@wangpatrick57
added all tests
bf8cd32
@wangpatrick57
wrote _count_files_in_workspace()
a6805a8
@wangpatrick57
comment
3dbeac0
@wangpatrick57
merged with main
2a838cb
@wangpatrick57 wangpatrick57 requested a review from lmwnshn July 8, 2024 00:08
@wangpatrick57 wangpatrick57 changed the title Clean workspace directory Automated workspace pruning/cleaning Jul 9, 2024
lmwnshn
lmwnshn reviewed Jul 14, 2024
View reviewed changes
manage/cli.py Outdated Show resolved Hide resolved
lmwnshn
lmwnshn approved these changes Jul 14, 2024
View reviewed changes
Copy link
Collaborator

@lmwnshn lmwnshn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wangpatrick57 wangpatrick57 merged commit 3245aab into cmu-db:main Jul 15, 2024
1 check passed
@wangpatrick57 wangpatrick57 deleted the clean-workspace branch July 18, 2024 00:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lmwnshn lmwnshn lmwnshn approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@wangpatrick57 @lmwnshn