Fine-tuning 🤖ChatGLM-6B model with 🤗PEFT.
👋 Join our WeChat.
[ English | 中文 ]
[23/04/20] Our repo achieved 100 stars within 12 days! Congratulations!
[23/04/19] Now we support merging the weights of fine-tuned models trained by LoRA! Try --checkpoint_dir checkpoint1,checkpoint2 argument for continually fine-tuning the models.
[23/04/18] Now we support training the quantized models using three fine-tuning methods! Try quantization_bit argument for training the model in 4/8 bits.
[23/04/12] Now we support training from checkpoints! Use --checkpoint_dir argument to specify the checkpoint model to fine-tune from.
[23/04/11] Now we support training with combined datasets! Try --dataset dataset1,dataset2 argument for training with multiple datasets.
Our script now supports the following datasets:
- Stanford Alpaca
- Stanford Alpaca (Chinese)
- GPT-4 Generated Data
- BELLE 2M
- BELLE 1M
- BELLE 0.5M
- BELLE Dialogue 0.4M
- BELLE School Math 0.25M
- BELLE Multiturn Chat 0.8M
- Guanaco Dataset
- Firefly 1.1M
- CodeAlpaca 20k
- Alpaca CoT
- Web QA (Chinese)
Please refer to data/README.md for details.
Some dataset requires confirmation before using it, so we recommend logging in with your HuggingFace account using these command.
pip install --upgrade huggingface_hub
huggingface-cli loginOur script now supports the following fine-tuning methods:
- LoRA
- Fine-tuning the low-rank adapters of the model.
- P-Tuning V2
- Fine-tuning the prefix encoder of the model.
- Freeze
- Fine-tuning the MLPs in the last n blocks of the model.
- Python 3.8+ and PyTorch 2.0.0
- 🤗Transformers, Datasets, Accelerate and PEFT (0.3.0.dev0 is required)
- protobuf, cpm_kernels, sentencepiece
- jieba, rouge_chinese, nltk
And powerful GPUs!
Please refer to data/example_dataset for checking the details about the format of dataset files. You can either use a single .json file or a dataset loading script with multiple files to create a custom dataset.
Note: please update data/dataset_info.json to use your custom dataset. About the format of this file, please refer to data/README.md.
git clone https://github.com/hiyouga/ChatGLM-Efficient-Tuning.git
conda create -n chatglm_etuning python=3.10
conda activate chatglm_etuning
cd ChatGLM-Efficient-Tuning
pip install -r requirements.txtCUDA_VISIBLE_DEVICES=0 python src/finetune.py \
--do_train \
--dataset alpaca_gpt4_zh \
--finetuning_type lora \
--output_dir path_to_checkpoint \
--per_device_train_batch_size 4 \
--gradient_accumulation_steps 4 \
--lr_scheduler_type cosine \
--logging_steps 10 \
--save_steps 1000 \
--learning_rate 5e-5 \
--num_train_epochs 1.0 \
--fp16Please refer to our Wiki about the details of the arguments.
accelerate config # configure the environment
accelerate launch python src/finetune.py # arguments (same as above)Note: if you are using LoRA method in fine-tuning, please provide --ddp_find_unused_parameters False argument to avoid the runtime error.
CUDA_VISIBLE_DEVICES=0 python src/finetune.py \
--do_eval \
--dataset alpaca_gpt4_zh \
--checkpoint_dir path_to_checkpoint \
--output_dir eval \
--per_device_eval_batch_size 8 \
--max_eval_samples 50 \
--predict_with_generateCUDA_VISIBLE_DEVICES=0 python src/infer.py \
--checkpoint_dir path_to_checkpointCUDA_VISIBLE_DEVICES=0 python src/web_demo.py \
--checkpoint_dir path_to_checkpointfrom .src import load_pretrained, ModelArguments
model_args = ModelArguments(checkpoint_dir=path_to_checkpoint_dir)
model, tokenizer = load_pretrained(model_args)
model = model.half().cuda()
# model.generate, model.chat()...| Fine-tune method | Batch size | Mode | GRAM | Speed |
|---|---|---|---|---|
| LoRA (r=8) | 16 | FP16 | 28GB | 8ex/s |
| LoRA (r=8) | 8 | FP16 | 24GB | 8ex/s |
| LoRA (r=8) | 4 | FP16 | 20GB | 8ex/s |
| LoRA (r=8) | 4 | INT8 | 10GB | 8ex/s |
| P-Tuning (p=16) | 4 | FP16 | 20GB | 8ex/s |
| P-Tuning (p=16) | 4 | INT8 | 16GB | 8ex/s |
| P-Tuning (p=16) | 4 | INT4 | 12GB | 8ex/s |
| Freeze (l=3) | 4 | FP16 | 24GB | 8ex/s |
| Freeze (l=3) | 4 | INT8 | 12GB | 8ex/s |
Note:
ris the lora rank,pis the number of prefix tokens,lis the number of trainable layers,ex/sis the examples per second in training. Thegradient_accumulation_stepsis set to1. All are evaluated on a single Tesla V100 (32G) GPU, they are approximated values and may vary in different GPUs.
We use the whole alpaca_gpt4_zh dataset to fine-tune the ChatGLM model with LoRA (r=8) for one epoch, using the default hyper-parameters. The loss curve during training is presented below.
We select 100 instances in the alpaca_gpt4_zh dataset to evaluate the fine-tuned ChatGLM model and compute the BLEU and ROUGE scores. The results are presented below.
| Score | Original | FZ (l=2) | PT (p=16) | LoRA (r=8) |
|---|---|---|---|---|
| BLEU-4 | 15.75 | 16.85 | 16.06 | 17.01 (+1.26) |
| Rouge-1 | 34.51 | 36.62 | 34.80 | 36.77 (+2.26) |
| Rouge-2 | 15.11 | 17.04 | 15.32 | 16.83 (+1.72) |
| Rouge-l | 26.18 | 28.17 | 26.35 | 28.86 (+2.68) |
| Params (%) | / | 4.35% | 0.06% | 0.06% |
FZ: freeze tuning, PT: P-Tuning V2 (we use
pre_seq_len=16for fair comparison with LoRA), Params: the percentange of trainable parameters.
- THUDM/ChatGLM-6B
- Official implementation of fine-tuning ChatGLM with P-Tuning v2 on the ADGEN dataset.
- Our fine-tuning script is largely depend on it. We further implement the LoRA tuning method. Additionally, we dynamically pad the inputs to the longest sequence in the batch instead of the maximum length, to accelerate the fine-tuning.
- mymusise/ChatGLM-Tuning
- An unoffical implementation of fine-tuning ChatGLM with LoRA on the Stanford Alpaca dataset.
- We borrowed some ideas from it. Our fine-tuning script integrates the data pre-processing part into the training procedure, so we need not generate a pre-processed dataset before training.
- ssbuild/chatglm_finetuning
- An unofficial implementation of fine-tuning ChatGLM with several PEFT methods on the Stanford Alpaca dataset.
- Our fine-tuning script is implemented purely with Huggingface transformers and is independent of the deep_training framework.
- lich99/ChatGLM-finetune-LoRA
- An unofficial implementation of fine-tuning ChatGLM with LoRA on the Stanford Alpaca dataset.
- We use the Huggingface PEFT to provide the state-of-the-art PEFT methods.
- liucongg/ChatGLM-Finetuning
- An unofficial implementation of fine-tuning ChatGLM with several methods including Freeze, LoRA and P-Tuning on the industrial dataset.
- We are aim to incorporate more instruction-following datasets for fine-tuning the ChatGLM model.
- yanqiangmiffy/InstructGLM
- An unofficial implementation of fine-tuning ChatGLM that explores the ChatGLM's ability on the instruction-following datasets.
- Our fine-tuning script integrates the data pre-processing part in to the training procedure.
- Employing LangChain to easily build applications that are capable of leveraging external knowledge upon fine-tuned ChatGLM models.
- Implementing the alignment algorithms to align human preferrences.
- Incorporating Chinese datasets into the training sets.
- Incorporating ChatGPT & GPT-4 self-chat data into the training sets.
- Implementing the Freeze-Tuning and P-Tuning method.
- Supporting Multi-GPUs fine-tuning.
- Adding script for evaluation. (but it appears very slow, increasing batch size may help)
- Loading from checkpoint.
- Fine-tuning the quantized model.
- Writing a guidebook about how to fine-tune ChatGLM with this framework.
- Combining with state-of-the-art model editing algorithms. (e.g. MEND)
- Incorporating the OpenAssistant Conversations Dataset for SFT and alignment.
This repository is licensed under the Apache-2.0 License. Please follow the Model License to use ChatGLM-6B model.
If this work is helpful, please cite as:
@Misc{chatglm-efficient-tuning,
title = {ChatGLM Efficient Tuning},
author = {hiyouga},
howpublished = {\url{https://github.com/hiyouga/ChatGLM-Efficient-Tuning}},
year = {2023}
}This repo benefits from ChatGLM-6B, ChatGLM-Tuning and yuanzhoulvpi2017/zero_nlp. Thanks for their wonderful works.