Skip to content

Commit

Permalink
fix: more robust check whether the HF model is quantized (#11891)
Browse files Browse the repository at this point in the history
Removes the check of `model.is_quantized` and adds more robust way of
checking for 4bit and 8bit quantization in the `huggingface_pipeline.py`
script. I had to make the original change on the outdated version of
`transformers`, because the models had this property before. Seems
redundant now.

Fixes: #11809 and
#11759
  • Loading branch information
eryk-dsai authored Oct 16, 2023
1 parent efa9ef7 commit 5019f59
Showing 1 changed file with 2 additions and 3 deletions.
5 changes: 2 additions & 3 deletions libs/langchain/langchain/llms/huggingface_pipeline.py
Original file line number Diff line number Diff line change
Expand Up @@ -109,9 +109,8 @@ def from_model_id(
) from e

if (
model.is_quantized
or model.model.is_loaded_in_4bit
or model.model.is_loaded_in_8bit
getattr(model, "is_loaded_in_4bit", False)
or getattr(model, "is_loaded_in_8bit", False)
) and device is not None:
logger.warning(
f"Setting the `device` argument to None from {device} to avoid "
Expand Down

0 comments on commit 5019f59

Please sign in to comment.