Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix evaluation if device not specified #1344

Merged
merged 1 commit into from
Apr 24, 2024
Merged

Conversation

awaelchli
Copy link
Contributor

@awaelchli awaelchli commented Apr 23, 2024

The command we have in the docs fails:

litgpt evaluate \
  --checkpoint_dir checkpoints/microsoft/phi-2/ \
  --batch_size 16 \
  --tasks "hellaswag" \
  --out_dir evaluate_model
Traceback (most recent call last):
  File "/home/zeus/miniconda3/envs/cloudspace/bin/litgpt", line 8, in <module>
    sys.exit(main())
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/litgpt/__main__.py", line 143, in main
    fn(**kwargs)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/litgpt/eval/evaluate.py", line 99, in convert_and_evaluate
    model = HFLM(repo_id, state_dict=state_dict, device=device, batch_size=batch_size, dtype=dtype)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lm_eval/models/huggingface.py", line 143, in __init__
    assert isinstance(device, str)
AssertionError

Because device=None is not accepted in lm_eval even though the type would suggest it is. This could be considered a bug, but we can choose a good default based on device availability.

@awaelchli awaelchli marked this pull request as ready for review April 23, 2024 22:17
Copy link
Collaborator

@rasbt rasbt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, I should have tested that when I added the code. Thanks for fixing!

@carmocca carmocca merged commit 9d032dd into main Apr 24, 2024
9 checks passed
@carmocca carmocca deleted the bugfix/fix-eval-device branch April 24, 2024 09:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants