Skip to content

Commit

Permalink
Add the huggingface token parameter, and modify the file path in llam…
Browse files Browse the repository at this point in the history
…a.cpp repo. (#761)

Signed-off-by: Song Liu <[email protected]>
Co-authored-by: Song Liu <[email protected]>
  • Loading branch information
melodyliu1986 and Song Liu authored Aug 19, 2024
1 parent f604294 commit e82e739
Show file tree
Hide file tree
Showing 4 changed files with 15 additions and 8 deletions.
4 changes: 2 additions & 2 deletions convert_models/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,10 @@ podman build -t converter .

## Quantize and Convert

You can run the conversion image directly with podman in the terminal. You just need to provide it with the huggingface model name you want to download, the quantization level you want to use and whether or not you want to keep the raw files after conversion.
You can run the conversion image directly with podman in the terminal. You just need to provide it with the huggingface model name you want to download, the quantization level you want to use and whether or not you want to keep the raw files after conversion. "HF_TOKEN" is optional, it is required for private models.

```bash
podman run -it --rm -v models:/converter/converted_models -e HF_MODEL_URL=<ORG/MODEL_NAME> -e QUANTIZATION=Q4_K_M -e KEEP_ORIGINAL_MODEL="False"
podman run -it --rm -v models:/converter/converted_models -e HF_MODEL_URL=<ORG/MODEL_NAME> -e HF_TOKEN=<YOUR_HF_TOKEN_ID> -e QUANTIZATION=Q4_K_M -e KEEP_ORIGINAL_MODEL="False" converter
```

You can also use the UI shown above to do the same.
Expand Down
2 changes: 2 additions & 0 deletions convert_models/download_huggingface.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,11 @@

parser = argparse.ArgumentParser()
parser.add_argument("-m", "--model")
parser.add_argument("-t", "--token")
args = parser.parse_args()

snapshot_download(repo_id=args.model,
token=args.token,
local_dir=f"converted_models/{args.model}",
local_dir_use_symlinks=True,
cache_dir=f"converted_models/cache")
9 changes: 5 additions & 4 deletions convert_models/run.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
#! /bin/bash

hf_model_url=${HF_MODEL_URL}
hf_token=${HF_TOKEN:="None"}
model_org=$(echo $hf_model_url | sed -n 's/\(.*\)\/\(.*\)/\1/p')
model_name=$(echo $hf_model_url | sed -n 's/\(.*\)\/\(.*\)/\2/p')
keep_orgi=${KEEP_ORIGINAL_MODEL}
Expand All @@ -15,11 +16,11 @@ if [ -e "/opt/app-root/src/converter/converted_models/cache/models--$model_org--
fi

echo "Downloading $hf_model_url"
python download_huggingface.py --model $hf_model_url
python llama.cpp/convert.py /opt/app-root/src/converter/converted_models/$hf_model_url
python llama.cpp/convert-hf-to-gguf.py /opt/app-root/src/converter/converted_models/$hf_model_url
python download_huggingface.py --model $hf_model_url --token $hf_token
python llama.cpp/examples/convert_legacy_llama.py /opt/app-root/src/converter/converted_models/$hf_model_url
python llama.cpp/convert_hf_to_gguf.py /opt/app-root/src/converter/converted_models/$hf_model_url
mkdir -p /opt/app-root/src/converter/converted_models/gguf/
llama.cpp/quantize /opt/app-root/src/converter/converted_models/$hf_model_url/ggml-model-f16.gguf /opt/app-root/src/converter/converted_models/gguf/$model_org-$model_name-${QUANTIZATION}.gguf ${QUANTIZATION}
llama.cpp/llama-quantize /opt/app-root/src/converter/converted_models/$hf_model_url/ggml-model-f16.gguf /opt/app-root/src/converter/converted_models/gguf/$model_org-$model_name-${QUANTIZATION}.gguf ${QUANTIZATION}
rm -rf /opt/app-root/src/converter/converted_models/$model_org

if [ $keep_orgi = "False" ]; then
Expand Down
8 changes: 6 additions & 2 deletions convert_models/ui.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@

model_name = st.text_input(label="Enter a huggingface model url to convert",
placeholder="org/model_name")
token_id = st.text_input(label="Enter your huggingface token (optional)",
help="huggingface token is required for private model"
) or "None"
keep_files = st.checkbox("Keep huggingface model files after conversion?")
submit_button = st.button(label="submit")
if submit_button:
Expand All @@ -30,8 +33,9 @@
"run",
"-it",
"--rm",
"-v", f"{volume}:/opt/app-root/src/converter/converted_models",
"-e", f"HF_MODEL_URL={model_name}" ,
"-v", f"{volume}:/converter/converted_models",
"-e", f"HF_MODEL_URL={model_name}",
"-e", f"HF_TOKEN={token_id}",
"-e", f"QUANTIZATION={quantization}",
"-e", f"KEEP_ORIGINAL_MODEL={keep_files}",
"converter"],stdout=subprocess.PIPE)
Expand Down

0 comments on commit e82e739

Please sign in to comment.