Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[1/2] Expand e2e testing to prepare for lm-eval #922

Merged
merged 14 commits into from
Nov 25, 2024
Merged

[1/2] Expand e2e testing to prepare for lm-eval #922

merged 14 commits into from
Nov 25, 2024

Conversation

dsikka
Copy link
Collaborator

@dsikka dsikka commented Nov 17, 2024 •
edited
Loading

Summary

  • Update e2e testing to prepare for lm_eval testing. Specifically, refactor the test to run oneshot through a utils function which can also be leveraged by lm_eval
  • Each test case now runs in its own process to better manage memory. This is now being managed by a bash script run_tests.sh
  • Update recipes to be better
  • Save recipes to the HFHub
  • Better handle session lifecycle
  • Update saving to the hub to save the entire directory

Running test cases

bash tests/e2e/vLLM/run_tests.sh

@github-actions GitHub Actions
Copy link

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

@dsikka dsikka changed the title update e2e testing; add lm_evl [1/2] Expand e2e testing for lm-eval Nov 17, 2024
@dsikka dsikka marked this pull request as ready for review November 17, 2024 22:44
@dsikka dsikka changed the title [1/2] Expand e2e testing for lm-eval [1/2] Expand e2e testing to prepare for lm-eval Nov 17, 2024
@dsikka dsikka added the ready When a PR is ready for review label Nov 17, 2024
kylesayrs
kylesayrs reviewed Nov 17, 2024
View reviewed changes
tests/e2e/e2e_utils.py Outdated Show resolved Hide resolved
@dsikka dsikka merged commit da12297 into main Nov 25, 2024
6 of 7 checks passed
@dsikka dsikka deleted the lm_eval_add branch November 25, 2024 16:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@horheynm horheynm horheynm approved these changes

@rahul-tuli rahul-tuli rahul-tuli approved these changes

@kylesayrs kylesayrs kylesayrs approved these changes

Assignees
No one assigned
Labels
ready When a PR is ready for review
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@dsikka @kylesayrs @rahul-tuli @horheynm