diff --git a/.gitignore b/.gitignore index dbb8acd6ab..a2e84c57ad 100644 --- a/.gitignore +++ b/.gitignore @@ -16,5 +16,3 @@ checkpoints out wandb events.out.tfevents* - -tests/reference_models diff --git a/README.md b/README.md index 1b7f307e44..6bca484f1a 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,5 @@
-LitGPT -  # ⚡ LitGPT @@ -18,29 +16,84 @@ Uses the latest state-of-the-art techniques: ![cpu-tests](https://github.com/lightning-AI/lit-stablelm/actions/workflows/cpu-tests.yml/badge.svg) [![license](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/Lightning-AI/lit-stablelm/blob/master/LICENSE) [![Discord](https://img.shields.io/discord/1077906959069626439)](https://discord.gg/VptPCZkGNa)

- Lightning.ai • - Install • - Get started • - Use LLMs • - Finetune, pretrain LLMs • + Lightning AI • Models • + Quick start • + Inference • + Finetune • + Pretrain • + Deploy • Features • Training recipes (YAML)

+  +LitGPT steps   -# Finetune, pretrain and deploy LLMs Lightning fast ⚡⚡ -LitGPT is a command-line tool designed to easily [finetune](#finetune-an-llm), [pretrain](#pretrain-an-llm), [evaluate](#use-an-llm), and deploy [20+ LLMs](#choose-from-20-llms) **on your own data**. It features highly-optimized [training recipes](#training-recipes) for the world's most powerful open-source large-language-models (LLMs). +# Finetune, pretrain and deploy LLMs Lightning fast ⚡⚡ +LitGPT is a command-line tool designed to easily [finetune](#finetune-an-llm), [pretrain](#pretrain-an-llm), [evaluate](#use-an-llm), and [deploy](#deploy-an-llm) [20+ LLMs](#choose-from-20-llms) **on your own data**. It features highly-optimized [training recipes](#training-recipes) for the world's most powerful open-source large language models (LLMs). -We reimplemented all model architectures and training recipes from scratch for 4 reasons: +We reimplemented all model architectures and training recipes from scratch for 4 reasons: -1. Remove all abstraction layers and have single file implementations. -2. Guarantee Apache 2.0 compliance to enable enterprise use without limits. -3. Optimized each model architectural detail to maximize performance, reduce costs, and speed up training. -4. Highly-optimized [recipe configs](#training-recipes) we have tested at enterprise scale. +1. Remove all abstraction layers and have single file implementations. +2. Guarantee Apache 2.0 compliance to enable enterprise use without limits. +3. Optimized each model's architectural detail to maximize performance, reduce costs, and speed up training. +4. Highly-optimized [recipe configs](#training-recipes) we have tested at enterprise scale. + +--- + +  + +# Choose from 20+ LLMs +LitGPT has 🤯 **custom, from-scratch implementations** of [20+ LLMs](tutorials/download_model_weights.md) without layers of abstraction: + +| Model | Model size | Author | Reference | +|----|----|----|----| +| Llama 3 | 8B, 70B | Meta AI | [Meta AI 2024](https://github.com/meta-llama/llama3) | +| Llama 2 | 7B, 13B, 70B | Meta AI | [Touvron et al. 2023](https://arxiv.org/abs/2307.09288) | +| Code Llama | 7B, 13B, 34B, 70B | Meta AI | [Rozière et al. 2023](https://arxiv.org/abs/2308.12950) | +| Mixtral MoE | 8x7B | Mistral AI | [Mistral AI 2023](https://mistral.ai/news/mixtral-of-experts/) | +| Mistral | 7B | Mistral AI | [Mistral AI 2023](https://mistral.ai/news/announcing-mistral-7b/) | +| CodeGemma | 7B | Google | [Google Team, Google Deepmind](https://ai.google.dev/gemma/docs/codegemma) | +| ... | ... | ... | ... | + +
+ See full list of 20+ LLMs + +  + +#### All models + +| Model | Model size | Author | Reference | +|----|----|----|----| +| CodeGemma | 7B | Google | [Google Team, Google Deepmind](https://ai.google.dev/gemma/docs/codegemma) | +| Code Llama | 7B, 13B, 34B, 70B | Meta AI | [Rozière et al. 2023](https://arxiv.org/abs/2308.12950) | +| Dolly | 3B, 7B, 12B | Databricks | [Conover et al. 2023](https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm) | +| Falcon | 7B, 40B, 180B | TII UAE | [TII 2023](https://falconllm.tii.ae) | +| FreeWilly2 (Stable Beluga 2) | 70B | Stability AI | [Stability AI 2023](https://stability.ai/blog/stable-beluga-large-instruction-fine-tuned-models) | +| Function Calling Llama 2 | 7B | Trelis | [Trelis et al. 2023](https://huggingface.co/Trelis/Llama-2-7b-chat-hf-function-calling-v2) | +| Gemma | 2B, 7B | Google | [Google Team, Google Deepmind](https://storage.googleapis.com/deepmind-media/gemma/gemma-report.pdf) | +| Llama 2 | 7B, 13B, 70B | Meta AI | [Touvron et al. 2023](https://arxiv.org/abs/2307.09288) | +| Llama 3 | 8B, 70B | Meta AI | [Meta AI 2024](https://github.com/meta-llama/llama3) | +| LongChat | 7B, 13B | LMSYS | [LongChat Team 2023](https://lmsys.org/blog/2023-06-29-longchat/) | +| Mixtral MoE | 8x7B | Mistral AI | [Mistral AI 2023](https://mistral.ai/news/mixtral-of-experts/) | +| Mistral | 7B | Mistral AI | [Mistral AI 2023](https://mistral.ai/news/announcing-mistral-7b/) | +| Nous-Hermes | 7B, 13B, 70B | NousResearch | [Org page](https://huggingface.co/NousResearch) | +| OpenLLaMA | 3B, 7B, 13B | OpenLM Research | [Geng & Liu 2023](https://github.com/openlm-research/open_llama) | +| Phi | 1.3B, 2.7B | Microsoft Research | [Li et al. 2023](https://arxiv.org/abs/2309.05463) | +| Platypus | 7B, 13B, 70B | Lee et al. | [Lee, Hunter, and Ruiz 2023](https://arxiv.org/abs/2308.07317) | +| Pythia | {14,31,70,160,410}M, {1,1.4,2.8,6.9,12}B | EleutherAI | [Biderman et al. 2023](https://arxiv.org/abs/2304.01373) | +| RedPajama-INCITE | 3B, 7B | Together | [Together 2023](https://together.ai/blog/redpajama-models-v1) | +| StableCode | 3B | Stability AI | [Stability AI 2023](https://stability.ai/blog/stablecode-llm-generative-ai-coding) | +| StableLM | 3B, 7B | Stability AI | [Stability AI 2023](https://github.com/Stability-AI/StableLM) | +| StableLM Zephyr | 3B | Stability AI | [Stability AI 2023](https://stability.ai/blog/stablecode-llm-generative-ai-coding) | +| TinyLlama | 1.1B | Zhang et al. | [Zhang et al. 2023](https://github.com/jzhang38/TinyLlama) | +| Vicuna | 7B, 13B, 33B | LMSYS | [Li et al. 2023](https://lmsys.org/blog/2023-03-30-vicuna/) + +
  @@ -66,34 +119,40 @@ pip install -e '.[all]' ``` -  - --- -# Get started -After installing LitGPT, select the model and action you want to take on that model (finetune, pretrain, evaluate, deploy, etc...): +  +# Quick start +After installing LitGPT, select the model and action you want to take on that model (finetune, pretrain, evaluate, deploy, etc...): ```bash # ligpt [action] [model] -litgpt download mistralai/Mistral-7B-Instruct-v0.2 -litgpt chat mistralai/Mistral-7B-Instruct-v0.2 -litgpt finetune mistralai/Mistral-7B-Instruct-v0.2 -litgpt pretrain mistralai/Mistral-7B-Instruct-v0.2 -litgpt serve mistralai/Mistral-7B-Instruct-v0.2 +litgpt download meta-llama/Meta-Llama-3-8B-Instruct +litgpt chat meta-llama/Meta-Llama-3-8B-Instruct +litgpt finetune meta-llama/Meta-Llama-3-8B-Instruct +litgpt pretrain meta-llama/Meta-Llama-3-8B-Instruct +litgpt serve meta-llama/Meta-Llama-3-8B-Instruct ```   -### Use an LLM -Here's an example showing how to use the Mistral 7B LLM. +### Use an LLM for inference +Use LLMs for inference to test its chatting capabilities, run evaluations, or extract embeddings, etc... +Here's an example showing how to use the Phi-2 LLM. + + + Open In Studio + + +  ```bash # 1) Download a pretrained model -litgpt download --repo_id mistralai/Mistral-7B-Instruct-v0.2 +litgpt download --repo_id microsoft/phi-2 # 2) Chat with the model litgpt chat \ - --checkpoint_dir checkpoints/mistralai/Mistral-7B-Instruct-v0.2 + --checkpoint_dir checkpoints/microsoft/phi-2 >> Prompt: What do Llamas eat? ``` @@ -105,28 +164,42 @@ For more information, refer to the [download](tutorials/download_model_weights.m ### Finetune an LLM [Finetune](tutorials/finetune.md) a model to specialize it on your own custom dataset: + + Open In Studio + + +  + ```bash # 1) Download a pretrained model litgpt download --repo_id microsoft/phi-2 # 2) Finetune the model -curl -L https://huggingface.co/datasets/medalpaca/medical_meadow_health_advice/raw/main/medical_meadow_health_advice.json -o my_custom_dataset.json +curl -L https://huggingface.co/datasets/ksaw008/finance_alpaca/resolve/main/finance_alpaca.json -o my_custom_dataset.json -litgpt finetune lora \ +litgpt finetune \ --checkpoint_dir checkpoints/microsoft/phi-2 \ --data JSON \ --data.json_path my_custom_dataset.json \ --data.val_split_fraction 0.1 \ - --out_dir out/phi-2-lora + --out_dir out/custom-model # 3) Chat with the model litgpt chat \ - --checkpoint_dir out/phi-2-lora/final + --checkpoint_dir out/custom-model/final ``` -### Pretrain an LLM +  + +### Pretrain an LLM Train an LLM from scratch on your own data via pretraining: + +Open In Studio + + +  + ```bash mkdir -p custom_texts curl https://www.gutenberg.org/cache/epub/24440/pg24440.txt --output custom_texts/book1.txt @@ -151,10 +224,19 @@ litgpt chat \ --checkpoint_dir out/custom-model/final ``` -### Continue pretraining an LLM -This is another way of finetuning that specialize an already pretrained model by training on custom data: +  -``` +### Continue pretraining an LLM +This is another way of finetuning that specializes an already pretrained model by training on custom data: + + + +Open In Studio + + +  + +```bash mkdir -p custom_texts curl https://www.gutenberg.org/cache/epub/24440/pg24440.txt --output custom_texts/book1.txt curl https://www.gutenberg.org/cache/epub/26393/pg26393.txt --output custom_texts/book2.txt @@ -165,6 +247,7 @@ litgpt download --repo_id EleutherAI/pythia-160m # 2) Continue pretraining the model litgpt pretrain \ --model_name pythia-160m \ + --tokenizer_dir checkpoints/EleutherAI/pythia-160m \ --initial_checkpoint_dir checkpoints/EleutherAI/pythia-160m \ --data TextFiles \ --data.train_data_path "custom_texts/" \ @@ -178,44 +261,45 @@ litgpt chat \   -> [!NOTE] -> **[Read the full docs](tutorials/0_to_litgpt.md)**. +### Deploy an LLM +Once you're ready to deploy a finetuned LLM, run this command: + + + Open In Studio +   ---- +```bash +# locate the checkpoint to your finetuned or pretrained model and call the `serve` command: +litgpt serve --checkpoint_dir path/to/your/checkpoint/microsoft/phi-2 -# Choose from 20+ LLMs +# Alternative: if you haven't finetuned, download any checkpoint to deploy it: +litgpt download --repo_id microsoft/phi-2 +litgpt serve --checkpoint_dir checkpoints/microsoft/phi-2 +``` -Use, Finetune, pretrain, deploy over 20+ LLMs ([full list](tutorials/download_model_weights.md)). +Test the server in a separate terminal and integrate the model API into your AI product: +```python +# 3) Use the server (in a separate session) +import requests, json + response = requests.post( + "http://127.0.0.1:8000/predict", + json={"prompt": "Fix typos in the following sentence: Exampel input"} +) +print(response.json()["output"]) +``` -| Model | Model size | Author | Reference | -|----|----|----|----| -| CodeGemma | 7B | Google | [Google Team, Google Deepmind](https://ai.google.dev/gemma/docs/codegemma) | -| Code Llama | 7B, 13B, 34B, 70B | Meta AI | [Rozière et al. 2023](https://arxiv.org/abs/2308.12950) | -| Dolly | 3B, 7B, 12B | Databricks | [Conover et al. 2023](https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm) | -| Falcon | 7B, 40B, 180B | TII UAE | [TII 2023](https://falconllm.tii.ae) | -| FreeWilly2 (Stable Beluga 2) | 70B | Stability AI | [Stability AI 2023](https://stability.ai/blog/stable-beluga-large-instruction-fine-tuned-models) | -| Function Calling Llama 2 | 7B | Trelis | [Trelis et al. 2023](https://huggingface.co/Trelis/Llama-2-7b-chat-hf-function-calling-v2) | -| Gemma | 2B, 7B | Google | [Google Team, Google Deepmind](https://storage.googleapis.com/deepmind-media/gemma/gemma-report.pdf) | -| Llama 2 | 7B, 13B, 70B | Meta AI | [Touvron et al. 2023](https://arxiv.org/abs/2307.09288) | -| LongChat | 7B, 13B | LMSYS | [LongChat Team 2023](https://lmsys.org/blog/2023-06-29-longchat/) | -| Mistral | 7B | Mistral AI | [Mistral website](https://mistral.ai/) | -| Nous-Hermes | 7B, 13B, 70B | NousResearch | [Org page](https://huggingface.co/NousResearch) | -| OpenLLaMA | 3B, 7B, 13B | OpenLM Research | [Geng & Liu 2023](https://github.com/openlm-research/open_llama) | -| Phi | 1.3B, 2.7B | Microsoft Research | [Li et al. 2023](https://arxiv.org/abs/2309.05463) | -| Platypus | 7B, 13B, 70B | Lee et al. | [Lee, Hunter, and Ruiz 2023](https://arxiv.org/abs/2308.07317) | -| Pythia | {14,31,70,160,410}M, {1,1.4,2.8,6.9,12}B | EleutherAI | [Biderman et al. 2023](https://arxiv.org/abs/2304.01373) | -| RedPajama-INCITE | 3B, 7B | Together | [Together 2023](https://together.ai/blog/redpajama-models-v1) | -| StableCode | 3B | Stability AI | [Stability AI 2023](https://stability.ai/blog/stablecode-llm-generative-ai-coding) | -| StableLM | 3B, 7B | Stability AI | [Stability AI 2023](https://github.com/Stability-AI/StableLM) | -| StableLM Zephyr | 3B | Stability AI | [Stability AI 2023](https://stability.ai/blog/stablecode-llm-generative-ai-coding) | -| TinyLlama | 1.1B | Zhang et al. | [Zhang et al. 2023](https://github.com/jzhang38/TinyLlama) | -| Vicuna | 7B, 13B, 33B | LMSYS | [Li et al. 2023](https://lmsys.org/blog/2023-03-30-vicuna/) +  + +> [!NOTE] +> **[Read the full docs](tutorials/0_to_litgpt.md)**.   -## State-of-the-art features +---- + +# State-of-the-art features ✅  State-of-the-art optimizations: Flash Attention v2, multi-GPU support via fully-sharded data parallelism, [optional CPU offloading](tutorials/oom.md#do-sharding-across-multiple-gpus), and [TPU and XLA support](extensions/xla). ✅  [Pretrain](tutorials/pretrain.md), [finetune](tutorials/finetune.md), and [deploy](tutorials/inference.md) @@ -247,7 +331,7 @@ Browse all training recipes [here](config_hub). ### Example ```bash -litgpt finetune lora \ +litgpt finetune \ --config https://raw.githubusercontent.com/Lightning-AI/litgpt/main/config_hub/finetune/llama-2-7b/lora.yaml ``` @@ -402,14 +486,14 @@ seed: 1337 Override any parameter in the CLI: ```bash -litgpt finetune lora \ +litgpt finetune \ --config https://raw.githubusercontent.com/Lightning-AI/litgpt/main/config_hub/finetune/llama-2-7b/lora.yaml \ --lora_r 4 ```   -# Community +# Community ## Get involved! diff --git a/config_hub/finetune/README.md b/config_hub/finetune/README.md index fc82e0854b..55b3d8d286 100644 --- a/config_hub/finetune/README.md +++ b/config_hub/finetune/README.md @@ -2,43 +2,71 @@ The table below lists the performances you can expect from the provided config files. Note that you can achieve lower memory consumption by lowering the micro batch size as needed. In addition, you can lower the rank (`lora_r`) in the LoRA configuration files and disable LoRA for certain layers (for example, setting `lora_projection` and other LoRA layer-specific parameters to `false`). For more information, see the [Dealing with out-of-memory (OOM) errors](../../tutorials/oom.md) on lowering the memory requirements. +The "Cost" column refers to the on-demand compute cost on [Lightning AI Studios where these benchmarks were executed](https://lightning.ai/lightning-ai/studios/automated-benchmarks-for-litgpt). +All experiments were conducted using bfloat-16 precision on the Alpaca2k dataset. The "Multitask score" refers to [MMLU](https://arxiv.org/abs/2009.03300).   -| | Size | Dataset | Epochs | Val loss | Peak memory | Max seq length | Micro batch size | Precision | Training runtime | -| --------------------------------- | ---- | --------- | ------ | -------- | ----------- | -------------- | ---------------- | --------- | -------------------| -| | | | | | | | | | | -| falcon-7b/lora.yaml | 7B | Alpaca 2k | 4 | 0.945 | 16.69 GB | 512 | 2 | bfloat16 | 24.88 min (1xA10G) | -| falcon-7b/qlora.yaml | 7B | Alpaca 2k | 4 | 0.993 | 9.44 GB | 512 | 2 | bfloat16 | 50.76 min (1xA10G) | -| | | | | | | | | | | -| gemma-2b/lora.yaml | 2B | Alpaca 2k | 2 | 1.476 | 12.62 GB | 512 | 2 | bfloat16 | 9.29 min (1xA10G) | -| gemma-2b/qlora.yaml | 2B | Alpaca 2k | 2 | 0.981 | 11.59 GB | 512 | 2 | bfloat16 | 12.90 min (1xA10G) | -| gemma-2b/full.yaml | 2B | Alpaca 2k | 0.35 | 0.990 | 17.43 GB | 512 | 1 | bfloat16 | 13.61 min (4xA10G) | -| | | | | | | | | | | -| gemma-7b/lora.yaml | 7B | Alpaca 2k | 2 | 0.903 | 25.30 GB | 512 | 1 | bfloat16 | 11.47 min (1xA100) | -| gemma-7b/qlora.yaml | 7B | Alpaca 2k | 2 | 0.951 | 17.31 GB | 512 | 1 | bfloat16 | 23.46 min (1xA100) | -| | | | | | | | | | | -| llama-2-7b/lora.yaml | 7B | Alpaca 2k | 4 | 0.802 | 19.77 GB | 512 | 2 | bfloat16 | 32.75 min (A10G) | -| llama-2-7b/qlora.yaml | 7B | Alpaca 2k | 4 | 0.814 | 13.68 GB | 512 | 2 | bfloat16 | 45.68 min (A10G) | -| llama-2-7b/full.yaml | 7B | Alpaca 2k | 1 | 0.941 | 26.81 GB | 512 | 4 | bfloat16 | 1.78 min (4xA100) | -| | | | | | | | | | | -| mistral-7b/lora.yaml (v0.1) | 7B | Alpaca 2k | 4 | 0.796 | 20.65 GB | 512 | 2 | bfloat16 | 31.04 min (1xA10G) | -| mistral-7b/qlora.yaml (v0.1) | 7B | Alpaca 2k | 4 | 0.803 | 14.29 GB | 512 | 2 | bfloat16 | 44.69 min (1xA10G) | -| | | | | | | | | | | -| mistral-7b-v0.2/lora.yaml | 7B | Alpaca 2k | 4 | 0.801 | 20.65 GB | 512 | 2 | bfloat16 | 30.96 min (1xA10G) | -| mistral-7b-v0.2/qlora.yaml | 7B | Alpaca 2k | 4 | 0.813 | 14.29 GB | 512 | 2 | bfloat16 | 44.68 min (1xA10G) | -| | | | | | | | | | | -| phi-2/lora.yaml | 2B | Alpaca 2k | 1 | 0.832 | 13.98 GB | 512 | 4 | bfloat16 | 3.82 min (1xA10G) | -| phi-2/qlora.yaml | 2B | Alpaca 2k | 1 | 0.846 | 14.27 GB | 512 | 4 | bfloat16 | 4.55 min (1xA10G) | -| phi-2/full.yaml | 2B | Alpaca 2k | 1 | 0.937 | 14.44 GB | 512 | 4 | bfloat16 | 13.00 min (1xA10G) | -| | | | | | | | | | | -| stablelm-base-alpha-3b/lora.yaml | 3B | Alpaca 2k | 4 | 1.367 | 8.58 GB | 512 | 2 | bfloat16 | 13.02 min (1xA10G) | -| stablelm-base-alpha-3b/qlora.yaml | 3B | Alpaca 2k | 4 | 1.392 | 5.24 GB | 512 | 2 | bfloat16 | 25.71 min (1xA10G) | -| stablelm-base-alpha-3b/full.yaml | 3B | Alpaca 2k | 1 | 1.494 | 21.23 GB | 512 | 1 | bfloat16 | 72.72 min (2xA10G) | -| | | | | | | | | | | -| tiny-llama/lora.yaml | 1.1B | Alpaca 2k | 3 | 1.038 | 13.50 GB | 512 | 8 | bfloat16 | 8.06 min (1xA10G) | -| tiny-llama/qlora.yaml | 1.1B | Alpaca 2k | 3 | 1.056 | 16.24 GB | 512 | 8 | bfloat16 | 8.74 min (1xA10G) | -| tiny-llama/full.yaml | 1.1B | Alpaca 2k | 1 | 1.105 | 14.10 GB | 512 | 4 | bfloat16 | 2.59 min (1xA10G) | +| Config | Model | Epochs | Max seq length | Micro batch size | Machine | Training runtime | Cost | Peak memory | Validation loss | Validation perplexity | Multitask score (MMLU) | +| --------------------------------- | ---------------------- | ------ | -------------- | ---------------- | ------- | ---------------- | ---- | ----------- | --------------- | --------------------- | --------------- | +| falcon-7b/lora.yaml | falcon-7b | 4 | 512 | 1 | 1xA10G | 24.84 min | $0.7 | 16.69 GB | 0.945 | 2.573 | 26.2% | +| falcon-7b/lora.yaml | falcon-7b | 4 | 512 | 1 | 4xA10G | 24.94 min | $2.0 | 16.69 GB | 0.945 | 2.573 | 26.4% | +| falcon-7b/qlora.yaml | falcon-7b | 4 | 512 | 1 | 1xA10G | 50.85 min | $1.5 | 9.44 GB | 0.993 | 2.699 | 26.3% | +| falcon-7b/qlora.yaml | falcon-7b | 4 | 512 | 1 | 4xA10G | 50.88 min | $4.1 | 9.44 GB | 0.993 | 2.699 | 26.3% | +| | | | | | | | | | | | | +| gemma-2b/full.yaml | gemma-2b | 1 | 512 | 1 | 4xA10G | 14.06 min | $1.1 | 17.43 GB | 1.021 | 2.777 | 32.4% | +| gemma-2b/lora.yaml | gemma-2b | 2 | 512 | 2 | 1xA10G | 9.41 min | $0.3 | 12.62 GB | 0.981 | 2.666 | 34.4% | +| gemma-2b/lora.yaml | gemma-2b | 2 | 512 | 2 | 4xA10G | 9.41 min | $0.8 | 12.62 GB | 0.981 | 2.667 | 34.0% | +| gemma-2b/qlora.yaml | gemma-2b | 2 | 512 | 2 | 1xA10G | 12.91 min | $0.4 | 11.58 GB | 1.085 | 2.959 | 36.4% | +| gemma-2b/qlora.yaml | gemma-2b | 2 | 512 | 2 | 4xA10G | 12.91 min | $1.0 | 11.59 GB | 1.085 | 2.958 | 36.4% | +| | | | | | | | | | | | | +| gemma-7b/lora.yaml | gemma-7b | 2 | 512 | 1 | 1xA10G | OOM | OOM | OOM | OOM | OOM | | +| gemma-7b/lora.yaml | gemma-7b | 2 | 512 | 1 | 4xA10G | OOM | OOM | OOM | OOM | OOM | | +| gemma-7b/qlora.yaml | gemma-7b | 2 | 512 | 1 | 1xA10G | 43.58 min | $1.3 | 17.18 GB | 0.973 | 2.646 | | +| gemma-7b/qlora.yaml | gemma-7b | 2 | 512 | 1 | 4xA10G | 43.58 min | $3.5 | 17.18 GB | 0.983 | 2.672 | | +| | | | | | | | | | | | | +| llama-2-7b/full.yaml | llama-2-7b | 1 | 512 | 4 | 4xA10G | OOM | OOM | OOM | OOM | OOM | | +| llama-2-7b/lora.yaml | llama-2-7b | 4 | 512 | 2 | 1xA10G | 32.82 min | $1.0 | 19.77 GB | 0.802 | 2.230 | 40.3% | +| llama-2-7b/lora.yaml | llama-2-7b | 4 | 512 | 2 | 4xA10G | 32.83 min | $2.6 | 19.77 GB | 0.802 | 2.229 | 40.2% | +| llama-2-7b/qlora.yaml | llama-2-7b | 4 | 512 | 2 | 1xA10G | 45.67 min | $1.4 | 13.68 GB | 0.814 | 2.258 | 38.6% | +| llama-2-7b/qlora.yaml | llama-2-7b | 4 | 512 | 2 | 4xA10G | 45.69 min | $3.7 | 13.68 GB | 0.815 | 2.258 | 38.6% | +| | | | | | | | | | | | | +| llama-3-8b/full.yaml | llama-3-8b | 1 | 512 | 4 | 4xA10G | OOM | OOM | OOM | OOM | OOM | | +| llama-3-8b/lora.yaml | llama-3-8b | 2 | 512 | 1 | 1xA10G | 14.79 min | $0.4 | 19.73 GB | 0.888 | 2.431 | 62.4% | +| llama-3-8b/lora.yaml | llama-3-8b | 2 | 512 | 1 | 4xA10G | 14.88 min | $1.2 | 19.73 GB | 0.889 | 2.432 | 62.5% | +| llama-3-8b/qlora.yaml | llama-3-8b | 2 | 512 | 2 | 1xA10G | 22.24 min | $0.7 | 17.41 GB | 0.939 | 2.558 | 62.2% | +| llama-3-8b/qlora.yaml | llama-3-8b | 2 | 512 | 2 | 4xA10G | 22.20 min | $1.8 | 17.41 GB | 0.939 | 2.557 | 62.2% | +| | | | | | | | | | | | | +| mistral-7b-v0.2/lora.yaml | mistral-7b-v0.2 | 4 | 512 | 2 | 1xA10G | 31.00 min | $0.9 | 20.66 GB | 0.801 | 2.228 | 55.7% | +| mistral-7b-v0.2/lora.yaml | mistral-7b-v0.2 | 4 | 512 | 2 | 4xA10G | 31.00 min | $2.5 | 20.66 GB | 0.802 | 2.229 | 55.5% | +| mistral-7b-v0.2/qlora.yaml | mistral-7b-v0.2 | 4 | 512 | 2 | 1xA10G | 44.75 min | $1.3 | 14.29 GB | 0.813 | 2.255 | 56.5% | +| mistral-7b-v0.2/qlora.yaml | mistral-7b-v0.2 | 4 | 512 | 2 | 4xA10G | 44.75 min | $3.6 | 14.29 GB | 0.813 | 2.254 | 56.3% | +| | | | | | | | | | | | | +| mistral-7b/lora.yaml | mistral-7b | 4 | 512 | 2 | 1xA10G | 31.01 min | $0.9 | 20.66 GB | 0.794 | 2.211 | 57.9% | +| mistral-7b/lora.yaml | mistral-7b | 4 | 512 | 2 | 4xA10G | 31.03 min | $2.5 | 20.66 GB | 0.796 | 2.218 | 57.9% | +| mistral-7b/qlora.yaml | mistral-7b | 4 | 512 | 2 | 1xA10G | 44.75 min | $1.3 | 14.29 GB | 0.803 | 2.231 | 57.9% | +| mistral-7b/qlora.yaml | mistral-7b | 4 | 512 | 2 | 4xA10G | 44.81 min | $3.6 | 14.29 GB | 0.803 | 2.233 | 57.6% | +| | | | | | | | | | | | | +| phi-2/full.yaml | phi-2 | 1 | 512 | 4 | 4xA10G | 11.87 min | $1.0 | 14.44 GB | 1.305 | 3.688 | 38.4% | +| phi-2/lora.yaml | phi-2 | 1 | 512 | 4 | 1xA10G | 3.78 min | $0.1 | 13.98 GB | 0.819 | 2.269 | 53.0% | +| phi-2/lora.yaml | phi-2 | 1 | 512 | 4 | 4xA10G | 3.78 min | $0.3 | 13.98 GB | 0.820 | 2.271 | 52.4% | +| phi-2/qlora.yaml | phi-2 | 1 | 512 | 4 | 1xA10G | 4.51 min | $0.1 | 14.27 GB | 0.837 | 2.310 | 52.3% | +| phi-2/qlora.yaml | phi-2 | 1 | 512 | 4 | 4xA10G | 4.52 min | $0.4 | 14.27 GB | 0.837 | 2.309 | 52.3% | +| | | | | | | | | | | | | +| stablelm-base-alpha-3b/full.yaml | stablelm-base-alpha-3b | 1 | 512 | 1 | 4xA10G | 70.13 min | $5.6 | 21.23 GB | 1.513 | 4.540 | 23.2% | +| stablelm-base-alpha-3b/lora.yaml | stablelm-base-alpha-3b | 4 | 512 | 1 | 1xA10G | 13.07 min | $0.4 | 8.58 GB | 1.361 | 3.900 | 25.9% | +| stablelm-base-alpha-3b/lora.yaml | stablelm-base-alpha-3b | 4 | 512 | 1 | 4xA10G | 13.16 min | $1.1 | 8.58 GB | 1.362 | 3.906 | 25.9% | +| stablelm-base-alpha-3b/qlora.yaml | stablelm-base-alpha-3b | 4 | 512 | 1 | 1xA10G | 25.86 min | $0.8 | 5.24 GB | 1.388 | 4.009 | 26.1% | +| stablelm-base-alpha-3b/qlora.yaml | stablelm-base-alpha-3b | 4 | 512 | 1 | 4xA10G | 25.80 min | $2.1 | 5.24 GB | 1.391 | 4.020 | 26.6% | +| | | | | | | | | | | | | +| tiny-llama/full.yaml | tiny-llama | 1 | 512 | 4 | 1xA10G | 2.58 min | $0.1 | 14.10 GB | 1.088 | 2.968 | 24.6% | +| tiny-llama/full.yaml | tiny-llama | 1 | 512 | 4 | 4xA10G | 2.57 min | $0.2 | 14.10 GB | 1.088 | 2.968 | 24.5% | +| tiny-llama/lora.yaml | tiny-llama | 3 | 512 | 8 | 1xA10G | 8.09 min | $0.2 | 13.50 GB | 1.039 | 2.826 | 25.5% | +| tiny-llama/qlora.yaml | tiny-llama | 3 | 512 | 8 | 1xA10G | 8.70 min | $0.3 | 16.24 GB | 1.056 | 2.874 | 25.3% | +| tiny-llama/qlora.yaml | tiny-llama | 3 | 512 | 8 | 4xA10G | 8.70 min | $0.7 | 16.24 GB | 1.056 | 2.874 | 25.4% | + +*OOM = Out of memory +   ## Extending the context length diff --git a/config_hub/finetune/falcon-7b/lora.yaml b/config_hub/finetune/falcon-7b/lora.yaml index eab0954182..c45b0fed94 100644 --- a/config_hub/finetune/falcon-7b/lora.yaml +++ b/config_hub/finetune/falcon-7b/lora.yaml @@ -114,6 +114,9 @@ eval: # Number of iterations (type: int, default: 100) max_iters: 100 + # Whether to evaluate on the validation set at the beginning of the training + initial_validation: false + # The name of the logger to send metrics to. (type: Literal['wandb', 'tensorboard', 'csv'], default: csv) logger_name: csv diff --git a/config_hub/finetune/falcon-7b/qlora.yaml b/config_hub/finetune/falcon-7b/qlora.yaml index dfc5377bd8..33ab9d9fc3 100644 --- a/config_hub/finetune/falcon-7b/qlora.yaml +++ b/config_hub/finetune/falcon-7b/qlora.yaml @@ -116,6 +116,9 @@ eval: # Number of iterations (type: int, default: 100) max_iters: 100 + # Whether to evaluate on the validation set at the beginning of the training + initial_validation: false + # The name of the logger to send metrics to. (type: Literal['wandb', 'tensorboard', 'csv'], default: csv) logger_name: csv diff --git a/config_hub/finetune/gemma-2b/full.yaml b/config_hub/finetune/gemma-2b/full.yaml index 77f20658ca..879f1afee9 100644 --- a/config_hub/finetune/gemma-2b/full.yaml +++ b/config_hub/finetune/gemma-2b/full.yaml @@ -85,6 +85,9 @@ eval: # Number of iterations (type: int, default: 100) max_iters: 100 + # Whether to evaluate on the validation set at the beginning of the training + initial_validation: false + # The name of the logger to send metrics to. (type: Literal['wandb', 'tensorboard', 'csv'], default: csv) logger_name: csv diff --git a/config_hub/finetune/gemma-2b/lora.yaml b/config_hub/finetune/gemma-2b/lora.yaml index c9f912a47c..91af82800d 100644 --- a/config_hub/finetune/gemma-2b/lora.yaml +++ b/config_hub/finetune/gemma-2b/lora.yaml @@ -115,6 +115,9 @@ eval: # Number of iterations (type: int, default: 100) max_iters: 100 + # Whether to evaluate on the validation set at the beginning of the training + initial_validation: false + # The name of the logger to send metrics to. (type: Literal['wandb', 'tensorboard', 'csv'], default: csv) logger_name: csv diff --git a/config_hub/finetune/gemma-2b/qlora.yaml b/config_hub/finetune/gemma-2b/qlora.yaml index dc15fe90d3..159ae2cc86 100644 --- a/config_hub/finetune/gemma-2b/qlora.yaml +++ b/config_hub/finetune/gemma-2b/qlora.yaml @@ -115,6 +115,9 @@ eval: # Number of iterations (type: int, default: 100) max_iters: 100 + # Whether to evaluate on the validation set at the beginning of the training + initial_validation: false + # The name of the logger to send metrics to. (type: Literal['wandb', 'tensorboard', 'csv'], default: csv) logger_name: csv diff --git a/config_hub/finetune/gemma-7b/lora.yaml b/config_hub/finetune/gemma-7b/lora.yaml index d7d56f5b5c..59120c5d0b 100644 --- a/config_hub/finetune/gemma-7b/lora.yaml +++ b/config_hub/finetune/gemma-7b/lora.yaml @@ -115,6 +115,9 @@ eval: # Number of iterations (type: int, default: 100) max_iters: 100 + # Whether to evaluate on the validation set at the beginning of the training + initial_validation: false + # The name of the logger to send metrics to. (type: Literal['wandb', 'tensorboard', 'csv'], default: csv) logger_name: csv diff --git a/config_hub/finetune/gemma-7b/qlora.yaml b/config_hub/finetune/gemma-7b/qlora.yaml index 7d4a2c634c..556fba0cf5 100644 --- a/config_hub/finetune/gemma-7b/qlora.yaml +++ b/config_hub/finetune/gemma-7b/qlora.yaml @@ -115,6 +115,9 @@ eval: # Number of iterations (type: int, default: 100) max_iters: 100 + # Whether to evaluate on the validation set at the beginning of the training + initial_validation: false + # The name of the logger to send metrics to. (type: Literal['wandb', 'tensorboard', 'csv'], default: csv) logger_name: csv diff --git a/config_hub/finetune/llama-2-7b/full.yaml b/config_hub/finetune/llama-2-7b/full.yaml index 10e439b2de..99de788c74 100644 --- a/config_hub/finetune/llama-2-7b/full.yaml +++ b/config_hub/finetune/llama-2-7b/full.yaml @@ -88,6 +88,9 @@ eval: # Number of iterations (type: int, default: 100) max_iters: 100 + # Whether to evaluate on the validation set at the beginning of the training + initial_validation: false + # The name of the logger to send metrics to. (type: Literal['wandb', 'tensorboard', 'csv'], default: csv) logger_name: csv diff --git a/config_hub/finetune/llama-2-7b/lora.yaml b/config_hub/finetune/llama-2-7b/lora.yaml index 91f326757a..594b2f924d 100644 --- a/config_hub/finetune/llama-2-7b/lora.yaml +++ b/config_hub/finetune/llama-2-7b/lora.yaml @@ -114,6 +114,9 @@ eval: # Number of iterations (type: int, default: 100) max_iters: 100 + # Whether to evaluate on the validation set at the beginning of the training + initial_validation: false + # The name of the logger to send metrics to. (type: Literal['wandb', 'tensorboard', 'csv'], default: csv) logger_name: csv diff --git a/config_hub/finetune/llama-2-7b/qlora.yaml b/config_hub/finetune/llama-2-7b/qlora.yaml index a3b7cb8dde..106b9422f4 100644 --- a/config_hub/finetune/llama-2-7b/qlora.yaml +++ b/config_hub/finetune/llama-2-7b/qlora.yaml @@ -116,6 +116,9 @@ eval: # Number of iterations (type: int, default: 100) max_iters: 100 + # Whether to evaluate on the validation set at the beginning of the training + initial_validation: false + # The name of the logger to send metrics to. (type: Literal['wandb', 'tensorboard', 'csv'], default: csv) logger_name: csv diff --git a/config_hub/finetune/llama-3-8b/full.yaml b/config_hub/finetune/llama-3-8b/full.yaml new file mode 100644 index 0000000000..e06d037710 --- /dev/null +++ b/config_hub/finetune/llama-3-8b/full.yaml @@ -0,0 +1,98 @@ + +# The path to the base model's checkpoint directory to load for finetuning. (type: , default: checkpoints/stabilityai/stablelm-base-alpha-3b) +checkpoint_dir: checkpoints/meta-llama/Meta-Llama-3-8B + +# Directory in which to save checkpoints and logs. (type: , default: out/finetune/full) +out_dir: out/finetune/full-llama-3-8b + +# The precision to use for finetuning. Possible choices: "bf16-true", "bf16-mixed", "32-true". (type: Optional[str], default: null) +precision: bf16-true + +# How many devices/GPUs to use (type: Union[int, str], default: 1) +devices: 4 + +# Path to a checkpoint directory to resume from in case training was interrupted, or ``True`` to resume +# from the latest checkpoint in ``out_dir``. (type: Union[bool, Path], default: False) +resume: false + +# Data-related arguments. If not provided, the default is ``litgpt.data.Alpaca``. +data: + class_path: litgpt.data.Alpaca2k + init_args: + mask_prompt: false + prompt_style: alpaca + ignore_index: -100 + seed: 42 + num_workers: 4 + +# Training-related arguments. See ``litgpt.args.TrainArgs`` for details +train: + + # Number of optimizer steps between saving checkpoints (type: Optional[int], default: 1000) + save_interval: 200 + + # Number of iterations between logging calls (type: int, default: 1) + log_interval: 1 + + # Number of samples between optimizer steps across data-parallel ranks (type: int, default: 64) + global_batch_size: 64 + + # Number of samples per data-parallel rank (type: int, default: 1) + micro_batch_size: 4 + + # Number of iterations with learning rate warmup active (type: int, default: 100) + lr_warmup_steps: 25 + + # Number of epochs to train on (type: Optional[int], default: 5) + epochs: 1 + + # Total number of tokens to train on (type: Optional[int], default: null) + max_tokens: + + # Limits the number of optimizer steps to run. (type: Optional[int], default: null) + max_steps: + + # Limits the length of samples. Off by default (type: Optional[int], default: null) + max_seq_length: 512 + + # Whether to tie the embedding weights with the language modeling head weights. (type: Optional[bool], default: null) + tie_embeddings: + + # (type: float, default: 0.003) + learning_rate: 0.0002 + + # (type: float, default: 0.02) + weight_decay: 0.1 + + # (type: float, default: 0.9) + beta1: 0.9 + + # (type: float, default: 0.95) + beta2: 0.95 + + # (type: Optional[float], default: null) + max_norm: + + # (type: float, default: 6e-05) + min_lr: 6.0e-05 + +# Evaluation-related arguments. See ``litgpt.args.EvalArgs`` for details +eval: + + # Number of optimizer steps between evaluation calls (type: int, default: 600) + interval: 25 + + # Number of tokens to generate (type: Optional[int], default: 100) + max_new_tokens: 100 + + # Number of iterations (type: int, default: 100) + max_iters: 100 + + # Whether to evaluate on the validation set at the beginning of the training + initial_validation: false + +# The name of the logger to send metrics to. (type: Literal['wandb', 'tensorboard', 'csv'], default: csv) +logger_name: csv + +# The random seed to use for reproducibility. (type: int, default: 1337) +seed: 1337 diff --git a/config_hub/finetune/llama-3-8b/lora.yaml b/config_hub/finetune/llama-3-8b/lora.yaml new file mode 100644 index 0000000000..1d874a0690 --- /dev/null +++ b/config_hub/finetune/llama-3-8b/lora.yaml @@ -0,0 +1,124 @@ + +# The path to the base model's checkpoint directory to load for finetuning. (type: , default: checkpoints/stabilityai/stablelm-base-alpha-3b) +checkpoint_dir: checkpoints/meta-llama/Meta-Llama-3-8B + +# Directory in which to save checkpoints and logs. (type: , default: out/lora) +out_dir: out/finetune/lora-llama-3-8b + +# The precision to use for finetuning. Possible choices: "bf16-true", "bf16-mixed", "32-true". (type: Optional[str], default: null) +precision: bf16-true + +# If set, quantize the model with this algorithm. See ``tutorials/quantize.md`` for more information. (type: Optional[Literal['nf4', 'nf4-dq', 'fp4', 'fp4-dq', 'int8-training']], default: null) +quantize: + +# How many devices/GPUs to use. (type: Union[int, str], default: 1) +devices: 1 + +# The LoRA rank. (type: int, default: 8) +lora_r: 32 + +# The LoRA alpha. (type: int, default: 16) +lora_alpha: 16 + +# The LoRA dropout value. (type: float, default: 0.05) +lora_dropout: 0.05 + +# Whether to apply LoRA to the query weights in attention. (type: bool, default: True) +lora_query: true + +# Whether to apply LoRA to the key weights in attention. (type: bool, default: False) +lora_key: false + +# Whether to apply LoRA to the value weights in attention. (type: bool, default: True) +lora_value: true + +# Whether to apply LoRA to the output projection in the attention block. (type: bool, default: False) +lora_projection: false + +# Whether to apply LoRA to the weights of the MLP in the attention block. (type: bool, default: False) +lora_mlp: false + +# Whether to apply LoRA to output head in GPT. (type: bool, default: False) +lora_head: false + +# Data-related arguments. If not provided, the default is ``litgpt.data.Alpaca``. +data: + class_path: litgpt.data.Alpaca2k + init_args: + mask_prompt: false + prompt_style: alpaca + ignore_index: -100 + seed: 42 + num_workers: 4 + +# Training-related arguments. See ``litgpt.args.TrainArgs`` for details +train: + + # Number of optimizer steps between saving checkpoints (type: Optional[int], default: 1000) + save_interval: 200 + + # Number of iterations between logging calls (type: int, default: 1) + log_interval: 1 + + # Number of samples between optimizer steps across data-parallel ranks (type: int, default: 128) + global_batch_size: 8 + + # Number of samples per data-parallel rank (type: int, default: 4) + micro_batch_size: 1 + + # Number of iterations with learning rate warmup active (type: int, default: 100) + lr_warmup_steps: 10 + + # Number of epochs to train on (type: Optional[int], default: 5) + epochs: 2 + + # Total number of tokens to train on (type: Optional[int], default: null) + max_tokens: + + # Limits the number of optimizer steps to run. (type: Optional[int], default: null) + max_steps: + + # Limits the length of samples. Off by default (type: Optional[int], default: null) + max_seq_length: 512 + + # Whether to tie the embedding weights with the language modeling head weights. (type: Optional[bool], default: null) + tie_embeddings: + + # (type: float, default: 0.0003) + learning_rate: 0.0002 + + # (type: float, default: 0.02) + weight_decay: 0.0 + + # (type: float, default: 0.9) + beta1: 0.9 + + # (type: float, default: 0.95) + beta2: 0.95 + + # (type: Optional[float], default: null) + max_norm: + + # (type: float, default: 6e-05) + min_lr: 6.0e-05 + +# Evaluation-related arguments. See ``litgpt.args.EvalArgs`` for details +eval: + + # Number of optimizer steps between evaluation calls (type: int, default: 100) + interval: 100 + + # Number of tokens to generate (type: Optional[int], default: 100) + max_new_tokens: 100 + + # Number of iterations (type: int, default: 100) + max_iters: 100 + + # Whether to evaluate on the validation set at the beginning of the training + initial_validation: false + +# The name of the logger to send metrics to. (type: Literal['wandb', 'tensorboard', 'csv'], default: csv) +logger_name: csv + +# The random seed to use for reproducibility. (type: int, default: 1337) +seed: 1337 diff --git a/config_hub/finetune/llama-3-8b/qlora.yaml b/config_hub/finetune/llama-3-8b/qlora.yaml new file mode 100644 index 0000000000..33a0fc98be --- /dev/null +++ b/config_hub/finetune/llama-3-8b/qlora.yaml @@ -0,0 +1,126 @@ + +# The path to the base model's checkpoint directory to load for finetuning. (type: , default: checkpoints/stabilityai/stablelm-base-alpha-3b) +checkpoint_dir: checkpoints/meta-llama/Meta-Llama-3-8B + +# Directory in which to save checkpoints and logs. (type: , default: out/lora) +out_dir: out/finetune/qlora-llama3-8b + +# The precision to use for finetuning. Possible choices: "bf16-true", "bf16-mixed", "32-true". (type: Optional[str], default: null) +precision: bf16-true + +# If set, quantize the model with this algorithm. See ``tutorials/quantize.md`` for more information. (type: Optional[Literal['nf4', 'nf4-dq', 'fp4', 'fp4-dq', 'int8-training']], default: null) +quantize: bnb.nf4 + +# How many devices/GPUs to use. (type: Union[int, str], default: 1) +devices: 1 + +# The LoRA rank. (type: int, default: 8) +lora_r: 32 + +# The LoRA alpha. (type: int, default: 16) +lora_alpha: 16 + +# The LoRA dropout value. (type: float, default: 0.05) +lora_dropout: 0.05 + +# Whether to apply LoRA to the query weights in attention. (type: bool, default: True) +lora_query: true + +# Whether to apply LoRA to the key weights in attention. (type: bool, default: False) +lora_key: false + +# Whether to apply LoRA to the value weights in attention. (type: bool, default: True) +lora_value: true + +# Whether to apply LoRA to the output projection in the attention block. (type: bool, default: False) +lora_projection: false + +# Whether to apply LoRA to the weights of the MLP in the attention block. (type: bool, default: False) +lora_mlp: false + +# Whether to apply LoRA to output head in GPT. (type: bool, default: False) +lora_head: false + +# Data-related arguments. If not provided, the default is ``litgpt.data.Alpaca``. +data: + class_path: litgpt.data.Alpaca2k + init_args: + mask_prompt: false + val_split_fraction: 0.05 + prompt_style: alpaca + ignore_index: -100 + seed: 42 + num_workers: 4 + download_dir: data/alpaca2k + +# Training-related arguments. See ``litgpt.args.TrainArgs`` for details +train: + + # Number of optimizer steps between saving checkpoints (type: Optional[int], default: 1000) + save_interval: 200 + + # Number of iterations between logging calls (type: int, default: 1) + log_interval: 1 + + # Number of samples between optimizer steps across data-parallel ranks (type: int, default: 128) + global_batch_size: 8 + + # Number of samples per data-parallel rank (type: int, default: 4) + micro_batch_size: 2 + + # Number of iterations with learning rate warmup active (type: int, default: 100) + lr_warmup_steps: 10 + + # Number of epochs to train on (type: Optional[int], default: 5) + epochs: 2 + + # Total number of tokens to train on (type: Optional[int], default: null) + max_tokens: + + # Limits the number of optimizer steps to run (type: Optional[int], default: null) + max_steps: + + # Limits the length of samples (type: Optional[int], default: null) + max_seq_length: 512 + + # Whether to tie the embedding weights with the language modeling head weights (type: Optional[bool], default: null) + tie_embeddings: + + # (type: float, default: 0.0003) + learning_rate: 0.0002 + + # (type: float, default: 0.02) + weight_decay: 0.0 + + # (type: float, default: 0.9) + beta1: 0.9 + + # (type: float, default: 0.95) + beta2: 0.95 + + # (type: Optional[float], default: null) + max_norm: + + # (type: float, default: 6e-05) + min_lr: 6.0e-05 + +# Evaluation-related arguments. See ``litgpt.args.EvalArgs`` for details +eval: + + # Number of optimizer steps between evaluation calls (type: int, default: 100) + interval: 100 + + # Number of tokens to generate (type: Optional[int], default: 100) + max_new_tokens: 100 + + # Number of iterations (type: int, default: 100) + max_iters: 100 + + # Whether to evaluate on the validation set at the beginning of the training + initial_validation: false + +# The name of the logger to send metrics to. (type: Literal['wandb', 'tensorboard', 'csv'], default: csv) +logger_name: csv + +# The random seed to use for reproducibility. (type: int, default: 1337) +seed: 1337 diff --git a/config_hub/finetune/mistral-7b-v0.2/lora.yaml b/config_hub/finetune/mistral-7b-v0.2/lora.yaml index aad8f7c986..f56e34c525 100644 --- a/config_hub/finetune/mistral-7b-v0.2/lora.yaml +++ b/config_hub/finetune/mistral-7b-v0.2/lora.yaml @@ -114,6 +114,9 @@ eval: # Number of iterations (type: int, default: 100) max_iters: 100 + # Whether to evaluate on the validation set at the beginning of the training + initial_validation: false + # The name of the logger to send metrics to. (type: Literal['wandb', 'tensorboard', 'csv'], default: csv) logger_name: csv diff --git a/config_hub/finetune/mistral-7b-v0.2/qlora.yaml b/config_hub/finetune/mistral-7b-v0.2/qlora.yaml index e2f5c3aafc..b648b24d72 100644 --- a/config_hub/finetune/mistral-7b-v0.2/qlora.yaml +++ b/config_hub/finetune/mistral-7b-v0.2/qlora.yaml @@ -116,6 +116,9 @@ eval: # Number of iterations (type: int, default: 100) max_iters: 100 + # Whether to evaluate on the validation set at the beginning of the training + initial_validation: false + # The name of the logger to send metrics to. (type: Literal['wandb', 'tensorboard', 'csv'], default: csv) logger_name: csv diff --git a/config_hub/finetune/mistral-7b/lora.yaml b/config_hub/finetune/mistral-7b/lora.yaml index adfed6b08d..e991ec424e 100644 --- a/config_hub/finetune/mistral-7b/lora.yaml +++ b/config_hub/finetune/mistral-7b/lora.yaml @@ -114,6 +114,9 @@ eval: # Number of iterations (type: int, default: 100) max_iters: 100 + # Whether to evaluate on the validation set at the beginning of the training + initial_validation: false + # The name of the logger to send metrics to. (type: Literal['wandb', 'tensorboard', 'csv'], default: csv) logger_name: csv diff --git a/config_hub/finetune/mistral-7b/qlora.yaml b/config_hub/finetune/mistral-7b/qlora.yaml index 7972048f46..e43b745bb8 100644 --- a/config_hub/finetune/mistral-7b/qlora.yaml +++ b/config_hub/finetune/mistral-7b/qlora.yaml @@ -116,6 +116,9 @@ eval: # Number of iterations (type: int, default: 100) max_iters: 100 + # Whether to evaluate on the validation set at the beginning of the training + initial_validation: false + # The name of the logger to send metrics to. (type: Literal['wandb', 'tensorboard', 'csv'], default: csv) logger_name: csv diff --git a/config_hub/finetune/phi-2/full.yaml b/config_hub/finetune/phi-2/full.yaml index 65040a393e..5b302a48ac 100644 --- a/config_hub/finetune/phi-2/full.yaml +++ b/config_hub/finetune/phi-2/full.yaml @@ -88,6 +88,9 @@ eval: # Number of iterations (type: int, default: 100) max_iters: 100 + # Whether to evaluate on the validation set at the beginning of the training + initial_validation: false + # The name of the logger to send metrics to. (type: Literal['wandb', 'tensorboard', 'csv'], default: csv) logger_name: csv diff --git a/config_hub/finetune/phi-2/lora.yaml b/config_hub/finetune/phi-2/lora.yaml index a3f348c8b2..2571bc02d0 100644 --- a/config_hub/finetune/phi-2/lora.yaml +++ b/config_hub/finetune/phi-2/lora.yaml @@ -115,6 +115,9 @@ eval: # Number of iterations (type: int, default: 100) max_iters: 100 + # Whether to evaluate on the validation set at the beginning of the training + initial_validation: false + # The name of the logger to send metrics to. (type: Literal['wandb', 'tensorboard', 'csv'], default: csv) logger_name: csv diff --git a/config_hub/finetune/phi-2/qlora.yaml b/config_hub/finetune/phi-2/qlora.yaml index aa2c36d40a..d48d910939 100644 --- a/config_hub/finetune/phi-2/qlora.yaml +++ b/config_hub/finetune/phi-2/qlora.yaml @@ -115,6 +115,9 @@ eval: # Number of iterations (type: int, default: 100) max_iters: 100 + # Whether to evaluate on the validation set at the beginning of the training + initial_validation: false + # The name of the logger to send metrics to. (type: Literal['wandb', 'tensorboard', 'csv'], default: csv) logger_name: csv diff --git a/config_hub/finetune/stablelm-base-alpha-3b/full.yaml b/config_hub/finetune/stablelm-base-alpha-3b/full.yaml index bd68af8714..c196fcc017 100644 --- a/config_hub/finetune/stablelm-base-alpha-3b/full.yaml +++ b/config_hub/finetune/stablelm-base-alpha-3b/full.yaml @@ -85,6 +85,9 @@ eval: # Number of iterations (type: int, default: 100) max_iters: 100 + # Whether to evaluate on the validation set at the beginning of the training + initial_validation: false + # The name of the logger to send metrics to. (type: Literal['wandb', 'tensorboard', 'csv'], default: csv) logger_name: csv diff --git a/config_hub/finetune/stablelm-base-alpha-3b/lora.yaml b/config_hub/finetune/stablelm-base-alpha-3b/lora.yaml index e674cc8419..6e52ea2175 100644 --- a/config_hub/finetune/stablelm-base-alpha-3b/lora.yaml +++ b/config_hub/finetune/stablelm-base-alpha-3b/lora.yaml @@ -114,6 +114,9 @@ eval: # Number of iterations (type: int, default: 100) max_iters: 100 + # Whether to evaluate on the validation set at the beginning of the training + initial_validation: false + # The name of the logger to send metrics to. (type: Literal['wandb', 'tensorboard', 'csv'], default: csv) logger_name: csv diff --git a/config_hub/finetune/stablelm-base-alpha-3b/qlora.yaml b/config_hub/finetune/stablelm-base-alpha-3b/qlora.yaml index 27b579cbd8..ebd2f098eb 100644 --- a/config_hub/finetune/stablelm-base-alpha-3b/qlora.yaml +++ b/config_hub/finetune/stablelm-base-alpha-3b/qlora.yaml @@ -116,6 +116,9 @@ eval: # Number of iterations (type: int, default: 100) max_iters: 100 + # Whether to evaluate on the validation set at the beginning of the training + initial_validation: false + # The name of the logger to send metrics to. (type: Literal['wandb', 'tensorboard', 'csv'], default: csv) logger_name: csv diff --git a/config_hub/finetune/tiny-llama/full.yaml b/config_hub/finetune/tiny-llama/full.yaml index 4bc09e460b..fe1d1ef99d 100644 --- a/config_hub/finetune/tiny-llama/full.yaml +++ b/config_hub/finetune/tiny-llama/full.yaml @@ -85,6 +85,9 @@ eval: # Number of iterations (type: int, default: 100) max_iters: 100 + # Whether to evaluate on the validation set at the beginning of the training + initial_validation: false + # The name of the logger to send metrics to. (type: Literal['wandb', 'tensorboard', 'csv'], default: csv) logger_name: csv diff --git a/config_hub/finetune/tiny-llama/lora.yaml b/config_hub/finetune/tiny-llama/lora.yaml index 4991900954..c42ff28ff3 100644 --- a/config_hub/finetune/tiny-llama/lora.yaml +++ b/config_hub/finetune/tiny-llama/lora.yaml @@ -115,6 +115,9 @@ eval: # Number of iterations (type: int, default: 100) max_iters: 100 + # Whether to evaluate on the validation set at the beginning of the training + initial_validation: false + # The name of the logger to send metrics to. (type: Literal['wandb', 'tensorboard', 'csv'], default: csv) logger_name: csv diff --git a/config_hub/finetune/tiny-llama/qlora.yaml b/config_hub/finetune/tiny-llama/qlora.yaml index 1e8cf20b8a..7e80e4d0ca 100644 --- a/config_hub/finetune/tiny-llama/qlora.yaml +++ b/config_hub/finetune/tiny-llama/qlora.yaml @@ -115,6 +115,9 @@ eval: # Number of iterations (type: int, default: 100) max_iters: 100 + # Whether to evaluate on the validation set at the beginning of the training + initial_validation: false + # The name of the logger to send metrics to. (type: Literal['wandb', 'tensorboard', 'csv'], default: csv) logger_name: csv diff --git a/config_hub/pretrain/debug.yaml b/config_hub/pretrain/debug.yaml index 77ad6b13ad..e89dda3cc9 100644 --- a/config_hub/pretrain/debug.yaml +++ b/config_hub/pretrain/debug.yaml @@ -11,6 +11,9 @@ model_config: # /teamspace/jobs//share. (type: , default: out/pretrain) out_dir: out/pretrain/debug +# The precision to use for pretraining. Possible choices: "bf16-true", "bf16-mixed", "32-true". (type: Optional[str], default: null) +precision: bf16-mixed + # Optional path to a checkpoint directory to initialize the model from. # Useful for continued pretraining. Mutually exclusive with ``resume``. (type: Optional[Path], default: null) initial_checkpoint_dir: @@ -85,6 +88,9 @@ eval: # Number of iterations (type: int, default: 100) max_iters: 100 + # Whether to evaluate on the validation set at the beginning of the training + initial_validation: false + # How many devices/GPUs to use. Uses all GPUs by default. (type: Union[int, str], default: auto) devices: auto diff --git a/config_hub/pretrain/tinyllama.yaml b/config_hub/pretrain/tinyllama.yaml index fe43b8b216..e2418a5b17 100644 --- a/config_hub/pretrain/tinyllama.yaml +++ b/config_hub/pretrain/tinyllama.yaml @@ -11,6 +11,9 @@ model_config: # /teamspace/jobs//share. (type: , default: out/pretrain) out_dir: out/pretrain/tiny-llama +# The precision to use for pretraining. Possible choices: "bf16-true", "bf16-mixed", "32-true". (type: Optional[str], default: null) +precision: bf16-mixed + # Optional path to a checkpoint directory to initialize the model from. # Useful for continued pretraining. Mutually exclusive with ``resume``. (type: Optional[Path], default: null) initial_checkpoint_dir: @@ -85,6 +88,9 @@ eval: # Number of iterations (type: int, default: 100) max_iters: 100 + # Whether to evaluate on the validation set at the beginning of the training + initial_validation: false + # How many devices/GPUs to use. Uses all GPUs by default. (type: Union[int, str], default: auto) devices: auto diff --git a/config_hub/pretrain/tinystories.yaml b/config_hub/pretrain/tinystories.yaml index b6d37209b1..8ed53a09d7 100644 --- a/config_hub/pretrain/tinystories.yaml +++ b/config_hub/pretrain/tinystories.yaml @@ -27,6 +27,9 @@ model_config: # /teamspace/jobs//share. (type: , default: out/pretrain) out_dir: out/pretrain/stories15M +# The precision to use for pretraining. Possible choices: "bf16-true", "bf16-mixed", "32-true". (type: Optional[str], default: null) +precision: bf16-mixed + # Optional path to a checkpoint directory to initialize the model from. # Useful for continued pretraining. Mutually exclusive with ``resume``. (type: Optional[Path], default: null) initial_checkpoint_dir: @@ -101,6 +104,9 @@ eval: # Number of iterations (type: int, default: 100) max_iters: 100 + # Whether to evaluate on the validation set at the beginning of the training + initial_validation: false + # How many devices/GPUs to use. Uses all GPUs by default. (type: Union[int, str], default: auto) devices: auto diff --git a/extensions/xla/generate/adapter.py b/extensions/xla/generate/adapter.py index 097abe4e56..f6fb0920d3 100644 --- a/extensions/xla/generate/adapter.py +++ b/extensions/xla/generate/adapter.py @@ -29,7 +29,7 @@ def setup( adapter_path: Path = Path("out/adapter/alpaca/lit_model_adapter_finetuned.pth"), checkpoint_dir: Path = Path("checkpoints/tiiuae/falcon-7b"), max_new_tokens: int = 100, - top_k: Optional[int] = 200, + top_k: Optional[int] = 50, temperature: float = 0.8, precision: str = "bf16-true", ) -> None: diff --git a/litgpt/__main__.py b/litgpt/__main__.py index 59d53ac904..821c1f5801 100644 --- a/litgpt/__main__.py +++ b/litgpt/__main__.py @@ -1,4 +1,5 @@ # Copyright Lightning AI. Licensed under the Apache License 2.0, see LICENSE file. +import sys from typing import TYPE_CHECKING, Any @@ -24,6 +25,8 @@ from litgpt.scripts.download import download_from_hub as download_fn from litgpt.scripts.merge_lora import merge_lora as merge_lora_fn from litgpt.eval.evaluate import convert_and_evaluate as evaluate_fn +from litgpt.deploy.serve import run_server as serve_fn + if TYPE_CHECKING: from jsonargparse import ArgumentParser @@ -39,6 +42,12 @@ def _new_parser(**kwargs: Any) -> "ArgumentParser": return parser +def _rewrite_argv_for_default_subcommand(parser_data: dict, command: str, subcommand: str) -> None: + """Rewrites the `sys.argv` such that `litgpt command` defaults to `litgpt command subcommand`.""" + if len(sys.argv) > 2 and sys.argv[1] == command and sys.argv[2] not in parser_data[command].keys(): + sys.argv.insert(2, subcommand) + + def main() -> None: parser_data = { "download": {"help": "Download weights or tokenizer data from the Hugging Face Hub.", "fn": download_fn}, @@ -80,6 +89,7 @@ def main() -> None: }, "merge_lora": {"help": "Merges the LoRA weights with the base model.", "fn": merge_lora_fn}, "evaluate": {"help": "Evaluate a model with the LM Evaluation Harness.", "fn": evaluate_fn}, + "serve": {"help": "Serve and deploy a model with LitServe.", "fn": serve_fn}, } from jsonargparse import set_config_read_mode, set_docstring_parse_options @@ -87,6 +97,8 @@ def main() -> None: set_docstring_parse_options(attribute_docstrings=True) set_config_read_mode(urls_enabled=True) + _rewrite_argv_for_default_subcommand(parser_data, "finetune", "lora") + root_parser = _new_parser(prog="litgpt") # register level 1 subcommands and level 2 subsubcommands. If there are more levels in the future we would want to diff --git a/litgpt/args.py b/litgpt/args.py index b227ffe3f6..7e277fe9e6 100644 --- a/litgpt/args.py +++ b/litgpt/args.py @@ -79,3 +79,5 @@ class EvalArgs: """Number of tokens to generate""" max_iters: int = 100 """Number of iterations""" + initial_validation: bool = False + """Whether to evaluate on the validation set at the beginning of the training""" diff --git a/litgpt/config.py b/litgpt/config.py index 0a4234222d..e03fa8ae34 100644 --- a/litgpt/config.py +++ b/litgpt/config.py @@ -836,6 +836,56 @@ def norm_class(self) -> Type: copy["name"] = c["name"].format(kind) copy["hf_config"]["name"] = c["hf_config"]["name"].format(kind) configs.append(copy) + + +############### +# Meta LLaMA 3 +############### +llama_3 = [ + # https://huggingface.co/meta-llama/Meta-Llama-3-8B/blob/main/config.json + dict( + name="Llama-3-8B{}", + hf_config=dict(org="meta-llama", name="Meta-Llama-3-8B{}"), + block_size=8192, + vocab_size=128000, + padded_vocab_size=128256, + n_layer=32, + n_head=32, + n_query_groups=8, + rotary_percentage=1.0, + parallel_residual=False, + bias=False, + norm_class_name="RMSNorm", + mlp_class_name="LLaMAMLP", + intermediate_size=14336, + rope_base=500000, + ), + # https://huggingface.co/meta-llama/Meta-Llama-3-70B/blob/main/config.json + dict( + name="Llama-3-70B{}", + hf_config=dict(org="meta-llama", name="Meta-Llama-3-70B{}"), + block_size=8192, + vocab_size=128000, + padded_vocab_size=128256, + n_layer=80, + n_head=64, + n_embd=8192, + n_query_groups=8, + rotary_percentage=1.0, + parallel_residual=False, + bias=False, + norm_class_name="RMSNorm", + mlp_class_name="LLaMAMLP", + intermediate_size=28672, + rope_base=500000, + ), +] +for c in llama_3: + for kind in ("", "-Instruct"): + copy = deepcopy(c) + copy["name"] = c["name"].format(kind) + copy["hf_config"]["name"] = c["hf_config"]["name"].format(kind) + configs.append(copy) ############### diff --git a/litgpt/deploy/serve.py b/litgpt/deploy/serve.py new file mode 100644 index 0000000000..9cd594230d --- /dev/null +++ b/litgpt/deploy/serve.py @@ -0,0 +1,149 @@ +# Copyright Lightning AI. Licensed under the Apache License 2.0, see LICENSE file. +from pathlib import Path +from typing import Dict, Any, Optional +from litgpt.utils import check_valid_checkpoint_dir + +import lightning as L +from lightning_utilities.core.imports import RequirementCache +import torch + + +from litgpt.model import GPT +from litgpt.config import Config +from litgpt.tokenizer import Tokenizer +from litgpt.generate.base import generate +from litgpt.prompts import load_prompt_style, has_prompt_style, PromptStyle +from litgpt.utils import load_checkpoint, CLI, get_default_supported_precision + + +_LITSERVE_AVAILABLE = RequirementCache("litserve") +if _LITSERVE_AVAILABLE: + from litserve import LitAPI, LitServer +else: + LitAPI, LitServer = object, object + + +class SimpleLitAPI(LitAPI): + def __init__(self, + checkpoint_dir: Path, + precision: Optional[str] = None, + temperature: float = 0.8, + top_k: int = 50, + max_new_tokens: int = 50) -> None: + + if not _LITSERVE_AVAILABLE: + raise ImportError(str(_LITSERVE_AVAILABLE)) + + super().__init__() + self.checkpoint_dir = checkpoint_dir + self.precision = precision + self.temperature = temperature + self.top_k = top_k + self.max_new_tokens = max_new_tokens + + def setup(self, device: str) -> None: + # Setup the model so it can be called in `predict`. + config = Config.from_file(self.checkpoint_dir / "model_config.yaml") + device = torch.device(device) + torch.set_float32_matmul_precision("high") + + precision = self.precision or get_default_supported_precision(training=False) + + fabric = L.Fabric( + accelerator=device.type, + devices=1 if device.type=="cpu" else [device.index], + precision=precision, + ) + checkpoint_path = self.checkpoint_dir / "lit_model.pth" + self.tokenizer = Tokenizer(self.checkpoint_dir) + self.prompt_style = ( + load_prompt_style(self.checkpoint_dir) + if has_prompt_style(self.checkpoint_dir) + else PromptStyle.from_config(config) + ) + with fabric.init_module(empty_init=True): + model = GPT(config) + with fabric.init_tensor(): + # enable the kv cache + model.set_kv_cache(batch_size=1) + model.eval() + + self.model = fabric.setup_module(model) + load_checkpoint(fabric, self.model, checkpoint_path) + self.device = fabric.device + + def decode_request(self, request: Dict[str, Any]) -> Any: + # Convert the request payload to your model input. + prompt = request["prompt"] + prompt = self.prompt_style.apply(prompt) + encoded = self.tokenizer.encode(prompt, device=self.device) + return encoded + + def predict(self, inputs: torch.Tensor) -> Any: + # Run the model on the input and return the output. + prompt_length = inputs.size(0) + max_returned_tokens = prompt_length + self.max_new_tokens + + y = generate( + self.model, + inputs, + max_returned_tokens, + temperature=self.temperature, + top_k=self.top_k, + eos_id=self.tokenizer.eos_id + ) + + for block in self.model.transformer.h: + block.attn.kv_cache.reset_parameters() + return y + + def encode_response(self, output: torch.Tensor) -> Dict[str, Any]: + # Convert the model output to a response payload. + decoded_output = self.tokenizer.decode(output) + return {"output": decoded_output} + + +def run_server( + checkpoint_dir: Path = Path("checkpoints"), + precision: Optional[str] = None, + temperature: float = 0.8, + top_k: int = 200, + max_new_tokens: int = 50, + devices: int = 1, + accelerator: str = "auto", + port: int = 8000 +) -> None: + """Serve a LitGPT model using LitServe + + Arguments: + checkpoint_dir: The checkpoint directory to load the model from. + precision: Optional precision setting to instantiate the model weights in. By default, this will + automatically be inferred from the metadata in the given ``checkpoint_dir`` directory. + temperature: Temperature setting for the text generation. Value above 1 increase randomness. + Values below 1 decrease randomness. + top_k: The size of the pool of potential next tokens. Values larger than 1 result in more novel + generated text but can also lead to more incoherent texts. + max_new_tokens: The number of generation steps to take. + devices: How many devices/GPUs to use. + accelerator: The type of accelerator to use. For example, "auto", "cuda", "cpu", or "mps". + The "auto" setting (default) chooses a GPU if available, and otherwise uses a CPU. + port: The network port number on which the model is configured to be served. + """ + check_valid_checkpoint_dir(checkpoint_dir, model_filename="lit_model.pth") + + server = LitServer( + SimpleLitAPI( + checkpoint_dir=checkpoint_dir, + precision=precision, + temperature=temperature, + top_k=top_k, + max_new_tokens=max_new_tokens, + ), + accelerator=accelerator, + devices=devices) + + server.run(port=port) + + +if __name__ == "__main__": + CLI(run_server) diff --git a/litgpt/eval/evaluate.py b/litgpt/eval/evaluate.py index 2a2e7f3a50..29791630dd 100644 --- a/litgpt/eval/evaluate.py +++ b/litgpt/eval/evaluate.py @@ -4,7 +4,6 @@ import os from pathlib import Path from typing import Optional, Union -import yaml import torch from litgpt.scripts.convert_lit_checkpoint import convert_lit_checkpoint @@ -28,7 +27,7 @@ def prepare_results(results, save_filepath, print_results=True): def convert_and_evaluate( checkpoint_dir: Path, tasks: Optional[str] = None, - out_dir: Optional[str] = None, + out_dir: Optional[Path] = None, force_conversion: bool = False, num_fewshot: Optional[int] = None, batch_size: int = 1, @@ -36,7 +35,7 @@ def convert_and_evaluate( dtype: Optional[Union[str, torch.dtype]] = None, limit: Optional[float] = None, seed: int = 1234, - save_filepath: Optional[str] = None, + save_filepath: Optional[Path] = None, ) -> None: """Convert a LitGPT model and run the LM Evaluation Harness @@ -46,9 +45,7 @@ def convert_and_evaluate( Saves to `checkpoint_dir`/evaluate by default. force_conversion: Set to `True` to reconvert the model and override an existing model.pth from a previous evaluation call. - tasks: CSV of task names to evaluate. - By default, the following tasks are used: - "hellaswag,truthfulqa_mc2,mmlu" + tasks: CSV of task names to evaluate. Example: "hellaswag,truthfulqa_mc2,mmlu" num_fewshot: Number of examples in few-shot context. batch_size: Batch size configuration. device: Device to use for evaluation, for example, "cuda" or "cuda:0". @@ -72,6 +69,9 @@ def convert_and_evaluate( ) return + if device is None: + device = "cuda" if torch.cuda.is_available() else "cpu" + checkpoint_dir = Path(checkpoint_dir) if out_dir is None: @@ -81,22 +81,22 @@ def convert_and_evaluate( out_dir.mkdir(parents=True, exist_ok=True) save_filepath = out_dir / Path("results.json") if save_filepath is None else Path(save_filepath) - config_filepath = checkpoint_dir/"model_config.yaml" - - with open(config_filepath, encoding="utf-8") as f: - config_dict = yaml.safe_load(f) - repo_id = f"{config_dict['hf_config']['org']}/{config_dict['hf_config']['name']}" - - copy_config_files(source_dir=checkpoint_dir, out_dir=out_dir) - model_path = out_dir / "model.pth" + model_path = out_dir / "pytorch_model.bin" if not model_path.exists() or force_conversion: + copy_config_files(source_dir=checkpoint_dir, out_dir=out_dir) convert_lit_checkpoint(checkpoint_dir=checkpoint_dir, output_dir=out_dir) + + # Hack: LitGPT's conversion doesn't save a pickle file that is compatible to be loaded with + # `torch.load(..., weights_only=True)`, which is a requirement in HFLM. + # So we're `torch.load`-ing and `torch.sav`-ing it again to work around this. + state_dict = torch.load(out_dir / "model.pth") + torch.save(state_dict, model_path) + os.remove(out_dir / "model.pth") from lm_eval.models.huggingface import HFLM - state_dict = torch.load(model_path) - model = HFLM(repo_id, state_dict=state_dict, device=device, batch_size=batch_size, dtype=dtype) + model = HFLM(pretrained=str(out_dir.resolve()), device=device, batch_size=batch_size, dtype=dtype) os.environ["TOKENIZERS_PARALLELISM"] = "false" diff --git a/litgpt/finetune/adapter.py b/litgpt/finetune/adapter.py index 9326793e2b..313d0ea8e7 100644 --- a/litgpt/finetune/adapter.py +++ b/litgpt/finetune/adapter.py @@ -29,6 +29,7 @@ chunked_cross_entropy, copy_config_files, get_default_supported_precision, + init_out_dir, load_checkpoint, num_parameters, parse_devices, @@ -61,7 +62,8 @@ def setup( Arguments: checkpoint_dir: The path to the base model's checkpoint directory to load for finetuning. - out_dir: Directory in which to save checkpoints and logs. + out_dir: Directory in which to save checkpoints and logs. If running in a Lightning Studio Job, look for it in + /teamspace/jobs//share. precision: The precision to use for finetuning. Possible choices: "bf16-true", "bf16-mixed", "32-true". quantize: If set, quantize the model with this algorithm. See ``tutorials/quantize.md`` for more information. devices: How many devices/GPUs to use. @@ -75,6 +77,7 @@ def setup( pprint(locals()) data = Alpaca() if data is None else data devices = parse_devices(devices) + out_dir = init_out_dir(out_dir) check_valid_checkpoint_dir(checkpoint_dir) config = Config.from_file(checkpoint_dir / "model_config.yaml") @@ -178,6 +181,12 @@ def main( if fabric.device.type == "cuda": fabric.print(f"Memory used: {torch.cuda.max_memory_allocated() / 1e9:.02f} GB") + # Final evaluation + val_loss = validate(fabric, model, val_dataloader, dataclasses.replace(eval, max_iters=len(val_dataloader))) + metrics = {"val_loss": val_loss, "val_ppl": math.exp(val_loss)} + fabric.log_dict(metrics) + fabric.print(f"Final evaluation | val loss: {val_loss.item():.3f} | val ppl: {math.exp(val_loss):.3f}") + # Save the final Adapter checkpoint at the end of training save_path = out_dir / "final" / "lit_model.pth.adapter" save_path.parent.mkdir(parents=True, exist_ok=True) @@ -211,7 +220,12 @@ def fit( f" {model.max_seq_length} and context length is {model.config.block_size}" ) - validate(fabric, model, val_dataloader, tokenizer, dataclasses.replace(eval, max_iters=2), data) # sanity check + if eval.initial_validation: + val_loss = validate(fabric, model, val_dataloader, dataclasses.replace(eval, max_iters=len(val_dataloader))) + val_loss = f"{val_loss:.3f}" + else: + validate(fabric, model, val_dataloader, dataclasses.replace(eval, max_iters=2)) # sanity check + val_loss = "n/a" train_iterator = CycleIterator(train_dataloader) throughput = ThroughputMonitor(fabric, window_size=50) @@ -223,7 +237,6 @@ def fit( iter_num = 0 total_lengths = 0 total_t0 = time.perf_counter() - val_loss = "n/a" while step_count < max_steps and train_iterator.epoch < train.epochs: iter_num += 1 @@ -278,7 +291,8 @@ def fit( if not is_accumulating and step_count % eval.interval == 0: t0 = time.perf_counter() - val_loss = validate(fabric, model, val_dataloader, tokenizer, eval, data) + val_loss = validate(fabric, model, val_dataloader, eval) + generate_example(fabric, model, tokenizer, eval, data) t1 = time.perf_counter() - t0 fabric.print(f"iter {iter_num}: val loss {val_loss.item():.4f}, val time: {t1 * 1000:.2f} ms") metrics = {"val_loss": val_loss, "val_ppl": math.exp(val_loss)} @@ -295,11 +309,8 @@ def fit( save_prompt_style(data.prompt_style, checkpoint_file.parent) -# the adapter "kv cache" cannot be initialized under `inference_mode` @torch.no_grad() -def validate( - fabric: L.Fabric, model: GPT, val_dataloader: DataLoader, tokenizer: Tokenizer, eval: EvalArgs, data: DataModule -) -> torch.Tensor: +def validate(fabric: L.Fabric, model: GPT, val_dataloader: DataLoader, eval: EvalArgs) -> torch.Tensor: fabric.print("Validating ...") model.eval() losses = torch.zeros(min(len(val_dataloader), eval.max_iters)) @@ -311,12 +322,19 @@ def validate( losses[k] = chunked_cross_entropy(logits[..., :-1, :], targets[..., 1:], chunk_size=0) val_loss = losses.mean() + model.train() + return val_loss + - # produce an example: +# the adapter "kv cache" cannot be initialized under `inference_mode` +@torch.no_grad() +def generate_example(fabric: L.Fabric, model: GPT, tokenizer: Tokenizer, eval: EvalArgs, data: DataModule): instruction = "Recommend a movie for me to watch during the weekend and explain the reason." fabric.print(instruction) prompt = data.prompt_style.apply(instruction) encoded = tokenizer.encode(prompt, device=fabric.device) + model.eval() + with fabric.init_tensor(): # do not set `max_seq_length=max_returned_token` because memory is not a concern here model.set_kv_cache(batch_size=1) @@ -324,12 +342,10 @@ def validate( model, encoded, max_returned_tokens=len(encoded) + eval.max_new_tokens, temperature=0.8, eos_id=tokenizer.eos_id ) model.clear_kv_cache() + model.train() output = tokenizer.decode(output) fabric.print(output) - model.train() - return val_loss - def get_lr_scheduler(optimizer, warmup_steps: int, max_steps: int): # linear warmup followed by cosine annealing diff --git a/litgpt/finetune/adapter_v2.py b/litgpt/finetune/adapter_v2.py index 3c4634e354..39b2a2d0e2 100644 --- a/litgpt/finetune/adapter_v2.py +++ b/litgpt/finetune/adapter_v2.py @@ -29,6 +29,7 @@ chunked_cross_entropy, copy_config_files, get_default_supported_precision, + init_out_dir, load_checkpoint, num_parameters, parse_devices, @@ -61,7 +62,8 @@ def setup( Arguments: checkpoint_dir: The path to the base model's checkpoint directory to load for finetuning. - out_dir: Directory in which to save checkpoints and logs. + out_dir: Directory in which to save checkpoints and logs. If running in a Lightning Studio Job, look for it in + /teamspace/jobs//share. precision: The precision to use for finetuning. Possible choices: "bf16-true", "bf16-mixed", "32-true". quantize: If set, quantize the model with this algorithm. See ``tutorials/quantize.md`` for more information. devices: How many devices/GPUs to use. @@ -75,6 +77,7 @@ def setup( pprint(locals()) data = Alpaca() if data is None else data devices = parse_devices(devices) + out_dir = init_out_dir(out_dir) check_valid_checkpoint_dir(checkpoint_dir) config = Config.from_file(checkpoint_dir / "model_config.yaml") @@ -178,6 +181,12 @@ def main( if fabric.device.type == "cuda": fabric.print(f"Memory used: {torch.cuda.max_memory_allocated() / 1e9:.02f} GB") + # Final evaluation + val_loss = validate(fabric, model, val_dataloader, dataclasses.replace(eval, max_iters=len(val_dataloader))) + metrics = {"val_loss": val_loss, "val_ppl": math.exp(val_loss)} + fabric.log_dict(metrics) + fabric.print(f"Final evaluation | val loss: {val_loss.item():.3f} | val ppl: {math.exp(val_loss):.3f}") + # Save the final Adapter checkpoint at the end of training save_path = out_dir / "final" / "lit_model.pth.adapter_v2" save_path.parent.mkdir(parents=True, exist_ok=True) @@ -211,7 +220,12 @@ def fit( f" {model.max_seq_length} and context length is {model.config.block_size}" ) - validate(fabric, model, val_dataloader, tokenizer, dataclasses.replace(eval, max_iters=2), data) # sanity check + if eval.initial_validation: + val_loss = validate(fabric, model, val_dataloader, dataclasses.replace(eval, max_iters=len(val_dataloader))) + val_loss = f"{val_loss:.3f}" + else: + validate(fabric, model, val_dataloader, dataclasses.replace(eval, max_iters=2)) # sanity check + val_loss = "n/a" train_iterator = CycleIterator(train_dataloader) throughput = ThroughputMonitor(fabric, window_size=50) @@ -223,7 +237,6 @@ def fit( iter_num = 0 total_lengths = 0 total_t0 = time.perf_counter() - val_loss = "n/a" while step_count < max_steps and train_iterator.epoch < train.epochs: iter_num += 1 @@ -278,7 +291,8 @@ def fit( if not is_accumulating and step_count % eval.interval == 0: t0 = time.perf_counter() - val_loss = validate(fabric, model, val_dataloader, tokenizer, eval, data) + val_loss = validate(fabric, model, val_dataloader, eval) + generate_example(fabric, model, tokenizer, eval, data) t1 = time.perf_counter() - t0 fabric.print(f"iter {iter_num}: val loss {val_loss.item():.4f}, val time: {t1 * 1000:.2f} ms") metrics = {"val_loss": val_loss, "val_ppl": math.exp(val_loss)} @@ -295,11 +309,8 @@ def fit( save_prompt_style(data.prompt_style, checkpoint_file.parent) -# the adapter "kv cache" cannot be initialized under `inference_mode` @torch.no_grad() -def validate( - fabric: L.Fabric, model: GPT, val_dataloader: DataLoader, tokenizer: Tokenizer, eval: EvalArgs, data: DataModule -) -> torch.Tensor: +def validate(fabric: L.Fabric, model: GPT, val_dataloader: DataLoader, eval: EvalArgs) -> torch.Tensor: fabric.print("Validating ...") model.eval() losses = torch.zeros(min(len(val_dataloader), eval.max_iters)) @@ -311,12 +322,19 @@ def validate( losses[k] = chunked_cross_entropy(logits[..., :-1, :], targets[..., 1:], chunk_size=0) val_loss = losses.mean() + model.train() + return val_loss + - # produce an example: +# the adapter "kv cache" cannot be initialized under `inference_mode` +@torch.no_grad() +def generate_example(fabric: L.Fabric, model: GPT, tokenizer: Tokenizer, eval: EvalArgs, data: DataModule): instruction = "Recommend a movie for me to watch during the weekend and explain the reason." fabric.print(instruction) prompt = data.prompt_style.apply(instruction) encoded = tokenizer.encode(prompt, device=fabric.device) + model.eval() + with fabric.init_tensor(): # do not set `max_seq_length=max_returned_token` because memory is not a concern here model.set_kv_cache(batch_size=1) @@ -324,12 +342,10 @@ def validate( model, encoded, max_returned_tokens=len(encoded) + eval.max_new_tokens, temperature=0.8, eos_id=tokenizer.eos_id ) model.clear_kv_cache() + model.train() output = tokenizer.decode(output) fabric.print(output) - model.train() - return val_loss - def get_lr_scheduler(optimizer, warmup_steps: int, max_steps: int): # linear warmup followed by cosine annealing diff --git a/litgpt/finetune/full.py b/litgpt/finetune/full.py index 3a2e2a7176..01db855189 100644 --- a/litgpt/finetune/full.py +++ b/litgpt/finetune/full.py @@ -28,6 +28,7 @@ copy_config_files, get_default_supported_precision, load_checkpoint, + init_out_dir, num_parameters, parse_devices, save_hyperparameters, @@ -59,7 +60,8 @@ def setup( Arguments: checkpoint_dir: The path to the base model's checkpoint directory to load for finetuning. - out_dir: Directory in which to save checkpoints and logs. + out_dir: Directory in which to save checkpoints and logs. If running in a Lightning Studio Job, look for it in + /teamspace/jobs//share. precision: The precision to use for finetuning. Possible choices: "bf16-true", "bf16-mixed", "32-true". devices: How many devices/GPUs to use resume: Path to a checkpoint directory to resume from in case training was interrupted, or ``True`` to resume @@ -74,6 +76,7 @@ def setup( pprint(locals()) data = Alpaca() if data is None else data devices = parse_devices(devices) + out_dir = init_out_dir(out_dir) check_valid_checkpoint_dir(checkpoint_dir) config = Config.from_file(checkpoint_dir / "model_config.yaml") @@ -150,6 +153,12 @@ def main( if fabric.device.type == "cuda": fabric.print(f"Memory used: {torch.cuda.max_memory_allocated() / 1e9:.02f} GB") + # Final evaluation + val_loss = validate(fabric, model, val_dataloader, dataclasses.replace(eval, max_iters=len(val_dataloader))) + metrics = {"val_loss": val_loss, "val_ppl": math.exp(val_loss)} + fabric.log_dict(metrics, step=state["iter_num"]) + fabric.print(f"Final evaluation | val loss: {val_loss.item():.3f} | val ppl: {math.exp(val_loss):.3f}") + # Save the final checkpoint at the end of training save_path = out_dir / "final" / "lit_model.pth" save_path.parent.mkdir(parents=True, exist_ok=True) @@ -185,7 +194,13 @@ def fit( f" {model.max_seq_length} and context length is {model.config.block_size}" ) - validate(fabric, model, val_dataloader, tokenizer, dataclasses.replace(eval, max_iters=2), data) # sanity check + if eval.initial_validation: + val_loss = validate(fabric, model, val_dataloader, dataclasses.replace(eval, max_iters=len(val_dataloader))) + val_loss = f"{val_loss:.3f}" + else: + validate(fabric, model, val_dataloader, dataclasses.replace(eval, max_iters=2)) # sanity check + val_loss = "n/a" + initial_iter = state["iter_num"] max_steps = train.max_steps or float("inf") train_iterator = CycleIterator(train_dataloader) @@ -207,7 +222,6 @@ def fit( fabric.device ) fabric.barrier() - val_loss = "n/a" while state["step_count"] < max_steps and train_iterator.epoch < train.epochs: state["iter_num"] += 1 @@ -258,7 +272,8 @@ def fit( if not is_accumulating and state["step_count"] % eval.interval == 0: t0 = time.perf_counter() - val_loss = validate(fabric, model, val_dataloader, tokenizer, eval, data) + val_loss = validate(fabric, model, val_dataloader, eval) + generate_example(fabric, model, tokenizer, eval, data) t1 = time.perf_counter() - t0 fabric.print(f"iter {state['iter_num']}: val loss {val_loss.item():.4f}, val time: {t1 * 1000:.2f} ms") metrics = {"val_loss": val_loss, "val_ppl": math.exp(val_loss)} @@ -277,9 +292,7 @@ def fit( # FSDP has issues with `inference_mode` @torch.no_grad() -def validate( - fabric: L.Fabric, model: GPT, val_dataloader: DataLoader, tokenizer: Tokenizer, eval: EvalArgs, data: DataModule -) -> torch.Tensor: +def validate(fabric: L.Fabric, model: GPT, val_dataloader: DataLoader, eval: EvalArgs) -> torch.Tensor: fabric.print("Validating ...") model.eval() losses = torch.zeros(min(len(val_dataloader), eval.max_iters)) @@ -291,12 +304,18 @@ def validate( losses[k] = chunked_cross_entropy(logits[..., :-1, :], targets[..., 1:], chunk_size=0) val_loss = losses.mean() + model.train() + return val_loss + - # produce an example: +@torch.no_grad() +def generate_example(fabric: L.Fabric, model: GPT, tokenizer: Tokenizer, eval: EvalArgs, data: DataModule): instruction = "Recommend a movie for me to watch during the weekend and explain the reason." fabric.print(instruction) prompt = data.prompt_style.apply(instruction) encoded = tokenizer.encode(prompt, device=fabric.device) + model.eval() + with fabric.init_tensor(): # do not set `max_seq_length=max_returned_token` because memory is not a concern here model.set_kv_cache(batch_size=1) @@ -304,12 +323,10 @@ def validate( model, encoded, max_returned_tokens=len(encoded) + eval.max_new_tokens, temperature=0.8, eos_id=tokenizer.eos_id ) model.clear_kv_cache() + model.train() output = tokenizer.decode(output) fabric.print(output) - model.train() - return val_loss - def get_lr_scheduler(optimizer, warmup_steps: int, max_steps: int): # linear warmup followed by cosine annealing diff --git a/litgpt/finetune/lora.py b/litgpt/finetune/lora.py index bb60b2d180..ae48bbc8fe 100644 --- a/litgpt/finetune/lora.py +++ b/litgpt/finetune/lora.py @@ -31,6 +31,7 @@ copy_config_files, get_default_supported_precision, load_checkpoint, + init_out_dir, num_parameters, parse_devices, save_hyperparameters, @@ -71,7 +72,8 @@ def setup( Arguments: checkpoint_dir: The path to the base model's checkpoint directory to load for finetuning. - out_dir: Directory in which to save checkpoints and logs. + out_dir: Directory in which to save checkpoints and logs. If running in a Lightning Studio Job, look for it in + /teamspace/jobs//share. precision: The precision to use for finetuning. Possible choices: "bf16-true", "bf16-mixed", "32-true". quantize: If set, quantize the model with this algorithm. See ``tutorials/quantize.md`` for more information. devices: How many devices/GPUs to use. @@ -94,6 +96,7 @@ def setup( pprint(locals()) data = Alpaca() if data is None else data devices = parse_devices(devices) + out_dir = init_out_dir(out_dir) check_valid_checkpoint_dir(checkpoint_dir) config = Config.from_file( @@ -208,6 +211,12 @@ def main( if fabric.device.type == "cuda": fabric.print(f"Memory used: {torch.cuda.max_memory_allocated() / 1e9:.02f} GB") + # Final evaluation + val_loss = validate(fabric, model, val_dataloader, dataclasses.replace(eval, max_iters=len(val_dataloader))) + metrics = {"val_loss": val_loss, "val_ppl": math.exp(val_loss)} + fabric.log_dict(metrics) + fabric.print(f"Final evaluation | val loss: {val_loss.item():.3f} | val ppl: {math.exp(val_loss):.3f}") + # Save the final LoRA checkpoint at the end of training save_path = out_dir / "final" / "lit_model.pth.lora" save_path.parent.mkdir(parents=True, exist_ok=True) @@ -242,7 +251,12 @@ def fit( f" {model.max_seq_length} and context length is {model.config.block_size}" ) - validate(fabric, model, val_dataloader, tokenizer, dataclasses.replace(eval, max_iters=2), data) # sanity check + if eval.initial_validation: + val_loss = validate(fabric, model, val_dataloader, dataclasses.replace(eval, max_iters=len(val_dataloader))) + val_loss = f"{val_loss:.3f}" + else: + validate(fabric, model, val_dataloader, dataclasses.replace(eval, max_iters=2)) # sanity check + val_loss = "n/a" train_iterator = CycleIterator(train_dataloader) throughput = ThroughputMonitor(fabric, window_size=50) @@ -254,7 +268,6 @@ def fit( iter_num = 0 total_lengths = 0 total_t0 = time.perf_counter() - val_loss = "n/a" while step_count < max_steps and train_iterator.epoch < train.epochs: iter_num += 1 @@ -309,7 +322,8 @@ def fit( if not is_accumulating and step_count % eval.interval == 0: t0 = time.perf_counter() - val_loss = validate(fabric, model, val_dataloader, tokenizer, eval, data) + val_loss = validate(fabric, model, val_dataloader, eval) + generate_example(fabric, model, tokenizer, eval, data) t1 = time.perf_counter() - t0 fabric.print(f"iter {iter_num}: val loss {val_loss.item():.4f}, val time: {t1 * 1000:.2f} ms") metrics = {"val_loss": val_loss, "val_ppl": math.exp(val_loss)} @@ -328,9 +342,7 @@ def fit( # FSDP has issues with `inference_mode` @torch.no_grad() -def validate( - fabric: L.Fabric, model: GPT, val_dataloader: DataLoader, tokenizer: Tokenizer, eval: EvalArgs, data: DataModule -) -> torch.Tensor: +def validate(fabric: L.Fabric, model: GPT, val_dataloader: DataLoader, eval: EvalArgs) -> torch.Tensor: fabric.print("Validating ...") model.eval() losses = torch.zeros(min(len(val_dataloader), eval.max_iters)) @@ -343,11 +355,18 @@ def validate( val_loss = losses.mean() - # produce an example: + model.train() + return val_loss + + +@torch.no_grad() +def generate_example(fabric: L.Fabric, model: GPT, tokenizer: Tokenizer, eval: EvalArgs, data: DataModule): instruction = "Recommend a movie for me to watch during the weekend and explain the reason." fabric.print(instruction) prompt = data.prompt_style.apply(instruction) encoded = tokenizer.encode(prompt, device=fabric.device) + model.eval() + with fabric.init_tensor(): # do not set `max_seq_length=max_returned_token` because memory is not a concern here model.set_kv_cache(batch_size=1) @@ -355,12 +374,10 @@ def validate( model, encoded, max_returned_tokens=len(encoded) + eval.max_new_tokens, temperature=0.8, eos_id=tokenizer.eos_id ) model.clear_kv_cache() + model.train() output = tokenizer.decode(output) fabric.print(output) - model.train() - return val_loss - def get_lr_scheduler(optimizer, warmup_steps: int, max_steps: int): # linear warmup followed by cosine annealing diff --git a/litgpt/generate/adapter.py b/litgpt/generate/adapter.py index 104b3e20b0..91ebd18397 100644 --- a/litgpt/generate/adapter.py +++ b/litgpt/generate/adapter.py @@ -23,7 +23,7 @@ def main( checkpoint_dir: Path = Path("checkpoints/stabilityai/stablelm-base-alpha-3b"), quantize: Optional[Literal["bnb.nf4", "bnb.nf4-dq", "bnb.fp4", "bnb.fp4-dq", "bnb.int8"]] = None, max_new_tokens: int = 100, - top_k: Optional[int] = 200, + top_k: Optional[int] = 50, temperature: float = 0.8, precision: Optional[str] = None, ) -> None: diff --git a/litgpt/generate/adapter_v2.py b/litgpt/generate/adapter_v2.py index c7aeee8a91..0d25092135 100644 --- a/litgpt/generate/adapter_v2.py +++ b/litgpt/generate/adapter_v2.py @@ -23,7 +23,7 @@ def main( checkpoint_dir: Path = Path("checkpoints/stabilityai/stablelm-base-alpha-3b"), quantize: Optional[Literal["bnb.nf4", "bnb.nf4-dq", "bnb.fp4", "bnb.fp4-dq", "bnb.int8"]] = None, max_new_tokens: int = 100, - top_k: Optional[int] = 200, + top_k: Optional[int] = 50, temperature: float = 0.8, precision: Optional[str] = None, ) -> None: diff --git a/litgpt/generate/base.py b/litgpt/generate/base.py index 6488717429..060604b43f 100644 --- a/litgpt/generate/base.py +++ b/litgpt/generate/base.py @@ -96,7 +96,7 @@ def main( *, num_samples: int = 1, max_new_tokens: int = 50, - top_k: Optional[int] = 200, + top_k: Optional[int] = 50, temperature: float = 0.8, checkpoint_dir: Path = Path("checkpoints/stabilityai/stablelm-base-alpha-3b"), quantize: Optional[Literal["bnb.nf4", "bnb.nf4-dq", "bnb.fp4", "bnb.fp4-dq", "bnb.int8"]] = None, diff --git a/litgpt/generate/full.py b/litgpt/generate/full.py index 608115a5e1..c570e8dd2e 100644 --- a/litgpt/generate/full.py +++ b/litgpt/generate/full.py @@ -22,7 +22,7 @@ def main( checkpoint_dir: Path = Path("checkpoints/stabilityai/stablelm-base-alpha-3b"), quantize: Optional[Literal["bnb.nf4", "bnb.nf4-dq", "bnb.fp4", "bnb.fp4-dq", "bnb.int8"]] = None, max_new_tokens: int = 100, - top_k: Optional[int] = 200, + top_k: Optional[int] = 50, temperature: float = 0.8, precision: Optional[str] = None, ) -> None: diff --git a/litgpt/generate/sequentially.py b/litgpt/generate/sequentially.py index f804c4cffc..9f006ab47f 100644 --- a/litgpt/generate/sequentially.py +++ b/litgpt/generate/sequentially.py @@ -116,7 +116,7 @@ def main( *, num_samples: int = 1, max_new_tokens: int = 50, - top_k: Optional[int] = 200, + top_k: Optional[int] = 50, temperature: float = 0.8, checkpoint_dir: Path = Path("checkpoints/mistralai/Mistral-7B-Instruct-v0.1"), quantize: Optional[Literal["bnb.nf4", "bnb.nf4-dq", "bnb.fp4", "bnb.fp4-dq"]] = None, diff --git a/litgpt/generate/tp.py b/litgpt/generate/tp.py index 5c56dd1c09..41492f75b2 100644 --- a/litgpt/generate/tp.py +++ b/litgpt/generate/tp.py @@ -94,7 +94,7 @@ def main( *, num_samples: int = 1, max_new_tokens: int = 50, - top_k: Optional[int] = 200, + top_k: Optional[int] = 50, temperature: float = 0.8, checkpoint_dir: Path = Path("checkpoints/stabilityai/stablelm-base-alpha-3b"), quantize: Optional[Literal["bnb.nf4", "bnb.nf4-dq", "bnb.fp4", "bnb.fp4-dq"]] = None, diff --git a/litgpt/lora.py b/litgpt/lora.py index 51fd66713d..8fee63cbb6 100644 --- a/litgpt/lora.py +++ b/litgpt/lora.py @@ -264,18 +264,22 @@ def __init__( total_qkv = q_per_kv + 2 head_size = out_features // (self.n_query_groups * total_qkv) ind = range(out_features) - self.lora_ind = [] + lora_ind = [] if enable_q: q_ind = [x for x in ind if (x // head_size) % total_qkv < total_qkv - 2] - self.lora_ind.extend(q_ind) + lora_ind.extend(q_ind) if enable_k: k_ind = [x for x in ind if (x // head_size) % total_qkv == total_qkv - 2] - self.lora_ind.extend(k_ind) + lora_ind.extend(k_ind) if enable_v: v_ind = [x for x in ind if (x // head_size) % total_qkv == total_qkv - 1] - self.lora_ind.extend(v_ind) + lora_ind.extend(v_ind) + self._lora_ind = torch.tensor(lora_ind) + self._lora_ind_cache = {self._lora_ind.device: self._lora_ind} self.reset_parameters() + + def zero_pad(self, x: torch.Tensor) -> torch.Tensor: """Properly pad weight updates with zeros. @@ -328,15 +332,19 @@ def zero_pad(self, x: torch.Tensor) -> torch.Tensor: # ⚬ enable_lora: [True, False, True] # Then x has embeddings_size of 256 (2 * 128 as enable_lora only for query and value, not keys) and expected # embeddings_size is 384 (self.linear.out_features), so that means that we need to pad from 256 to 384 with zeros, but - # only for key updates (this is where self.lora_ind comes in handy) + # only for key updates (this is where lora_ind comes in handy) # Note: double transpose (in the beginning and in the end) is basically a guard for two-dimensional tensors # for example when we want to merge/unmerge LoRA weights and pretrained weights x = x.transpose(0, 1) result = x.new_zeros((*x.shape[:-1], self.linear.out_features)) # (64, 64, 384) result = result.view(-1, self.linear.out_features) # (4096, 384) - result = result.index_copy( - 1, torch.tensor(self.lora_ind, device=result.device), x.reshape(-1, sum(self.qkv_shapes)) - ) # (4096, 256) + + # `lora_ind` is constant, so we want to avoid copying it (and incurring an expensive cudaStreamSynchronize) + # every time this method is called. So instead we simply cache a copy on each device that needs it. + if (lora_ind := self._lora_ind_cache.get(result.device)) is None: + self._lora_ind_cache[result.device] = lora_ind = self._lora_ind.to(result.device) + + result = result.index_copy(1, lora_ind, x.reshape(-1, sum(self.qkv_shapes))) # (4096, 256) return result.view((*x.shape[:-1], self.linear.out_features)).transpose(0, 1) # (64, 64, 384) def conv1d(self, input: torch.Tensor, weight: torch.Tensor) -> torch.Tensor: diff --git a/litgpt/pretrain.py b/litgpt/pretrain.py index f75a93d8c6..d5014dc022 100644 --- a/litgpt/pretrain.py +++ b/litgpt/pretrain.py @@ -1,7 +1,6 @@ # Copyright Lightning AI. Licensed under the Apache License 2.0, see LICENSE file. import math -import os import pprint import time from datetime import timedelta @@ -30,6 +29,8 @@ choose_logger, chunked_cross_entropy, copy_config_files, + get_default_supported_precision, + init_out_dir, num_parameters, parse_devices, reset_parameters, @@ -42,6 +43,7 @@ def setup( model_name: Optional[str] = None, model_config: Optional[Config] = None, out_dir: Path = Path("out/pretrain"), + precision: Literal["bf16-true", "bf16-mixed", "32-true", None] = None, initial_checkpoint_dir: Optional[Path] = None, resume: Union[bool, Path] = False, data: Optional[DataModule] = None, @@ -75,6 +77,7 @@ def setup( ``model_config``. out_dir: Directory in which to save checkpoints and logs. If running in a Lightning Studio Job, look for it in /teamspace/jobs//share. + precision: The precision to use for finetuning. Determines a compatible precision setting by default. initial_checkpoint_dir: Optional path to a checkpoint directory to initialize the model from. Useful for continued pretraining. Mutually exclusive with ``resume``. resume: Path to a checkpoint directory to resume from in case training was interrupted, or ``True`` to resume @@ -96,6 +99,7 @@ def setup( available_models = "\n".join(sorted(name_to_config)) raise ValueError(f"Please specify --model_name . Available values:\n{available_models}") config = Config.from_name(model_name) if model_config is None else model_config + precision = precision or get_default_supported_precision(training=True) devices = parse_devices(devices) out_dir = init_out_dir(out_dir) # in case the dataset requires the Tokenizer @@ -109,7 +113,7 @@ def setup( strategy = FSDPStrategy(auto_wrap_policy={Block}, state_dict_type="full", sharding_strategy="HYBRID_SHARD") else: strategy = "auto" - fabric = L.Fabric(devices=devices, strategy=strategy, precision="bf16-mixed", loggers=[logger]) + fabric = L.Fabric(devices=devices, strategy=strategy, precision=precision, loggers=[logger]) fabric.launch() fabric.print(pprint.pformat(hparams)) @@ -169,12 +173,13 @@ def main( model = torch.compile(model) model = fabric.setup(model) + optimizer = torch.optim.AdamW( model.parameters(), lr=train.learning_rate, weight_decay=train.weight_decay, betas=(train.beta1, train.beta2), - fused=True, + fused=fabric.device.type == "cuda", ) optimizer = fabric.setup_optimizers(optimizer) @@ -223,7 +228,13 @@ def fit( model = state["model"] optimizer = state["optimizer"] - validate(fabric, model, val_dataloader, max_iters=2) # sanity check + if eval.initial_validation: + val_loss = validate(fabric, model, val_dataloader, max_iters=eval.max_iters) + val_loss = f"{val_loss:.3f}" + else: + validate(fabric, model, val_dataloader, max_iters=2) # sanity check + val_loss = "n/a" + throughput = ThroughputMonitor(fabric, window_size=5) with torch.device("meta"): @@ -247,7 +258,6 @@ def fit( ) fabric.barrier() total_t0 = time.perf_counter() - val_loss = "n/a" warmup_iters = train.warmup_iters(devices, max_iters, train_dataloader) @@ -404,12 +414,6 @@ def init_weights(module, std): reset_parameters(model) -def init_out_dir(out_dir: Path) -> Path: - if not out_dir.is_absolute() and "LIGHTNING_ARTIFACTS_DIR" in os.environ: - return Path(os.getenv("LIGHTNING_ARTIFACTS_DIR")) / out_dir - return out_dir - - def save_checkpoint(fabric, state, tokenizer_dir, checkpoint_file): model = state["model"] checkpoint_file.parent.mkdir(parents=True, exist_ok=True) diff --git a/litgpt/prompts.py b/litgpt/prompts.py index d827413913..04a0551cd1 100644 --- a/litgpt/prompts.py +++ b/litgpt/prompts.py @@ -200,6 +200,24 @@ def apply(self, prompt: str, **kwargs: str) -> str: ) +class Llama3(PromptStyle): + def apply(self, prompt: str, **kwargs: str) -> str: + # https://github.com/meta-llama/llama3/blob/359887376f0aaf30e433f23e25df858d8c2a9833/llama/tokenizer.py#L202-L229 + return ( + "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n" + "You are a helpful assistant.<|eot_id|>\n" # The system prompt is optional + "<|start_header_id|>user<|end_header_id|>\n\n" + f"{prompt}<|eot_id|>\n" + "<|start_header_id|>assistant<|end_header_id|>\n\n" + ) + + def stop_tokens(self, tokenizer: "Tokenizer") -> Tuple[List[int], ...]: + return ( + [tokenizer.eos_id], + [tokenizer.token_to_id("<|eot_id|>")], + ) + + class FreeWilly2(PromptStyle): def apply(self, prompt: str, **kwargs: str) -> str: return ( @@ -316,6 +334,8 @@ def model_name_to_prompt_style(model_name: str) -> PromptStyle: return Llama2FunctionCalling() if re.search("Llama-2.*-chat", model_name): return Llama2() + if re.search("Llama-3.*-Instruct", model_name): + return Llama3() if re.search("FreeWilly2", model_name): return FreeWilly2() if re.search("Platypus", model_name): diff --git a/litgpt/tokenizer.py b/litgpt/tokenizer.py index 55c972e69a..8217fcd069 100644 --- a/litgpt/tokenizer.py +++ b/litgpt/tokenizer.py @@ -73,11 +73,11 @@ def check_if_bos_token_used(self, checkpoint_dir: Path) -> bool: return False with open(tokenizer_config_path, encoding="utf-8") as fp: config = json.load(fp) - if any(config.get(check, False) for check in ("add_bos_token", "add_prefix_space")): - return True - # for examples that also use the Llama tokenizer, but do not have or set add_bos_token to True. + if "add_bos_token" in config: + return config["add_bos_token"] + # if `add_bos_token` isn't in the config file, but LLaMA tokenizer is used - return True. # ex: https://huggingface.co/stabilityai/StableBeluga2/blob/main/tokenizer_config.json#L2 - return config.get("add_bos_token") is None and config.get("tokenizer_class") == "LlamaTokenizer" + return config.get("tokenizer_class") == "LlamaTokenizer" def encode( self, diff --git a/litgpt/utils.py b/litgpt/utils.py index 0e40d336a1..6eb7efbff4 100644 --- a/litgpt/utils.py +++ b/litgpt/utils.py @@ -3,6 +3,7 @@ """Utility functions for training and inference.""" import inspect import math +import os import pickle import shutil import sys @@ -27,6 +28,12 @@ from litgpt import GPT, Config +def init_out_dir(out_dir: Path) -> Path: + if not out_dir.is_absolute() and "LIGHTNING_ARTIFACTS_DIR" in os.environ: + return Path(os.getenv("LIGHTNING_ARTIFACTS_DIR")) / out_dir + return out_dir + + def find_multiple(n: int, k: int) -> int: assert k > 0 if n % k == 0: @@ -265,7 +272,8 @@ def chunked_cross_entropy( for logit_chunk, target_chunk in zip(logit_chunks, target_chunks) ] non_masked_elems = (targets != ignore_index).sum() - return torch.cat(loss_chunks).sum() / max(1, non_masked_elems) + # See [non_masked_elems div note] + return torch.cat(loss_chunks).sum() / non_masked_elems.maximum(torch.ones_like(non_masked_elems)) # no chunking at all logits = logits.reshape(-1, logits.size(-1)) @@ -281,7 +289,11 @@ def chunked_cross_entropy( for logit_chunk, target_chunk in zip(logit_chunks, target_chunks) ] non_masked_elems = (targets != ignore_index).sum() - return torch.cat(loss_chunks).sum() / max(1, non_masked_elems) + # [non_masked_elems div note]: + # max(1, non_masked_elems) would be more ergonomic to avoid a division by zero. However that + # results in a python int which is then passed back to torch division. By using the + # `x.maximum(torch.ones_like(x))` pattern we avoid a cudaStreamSynchronize. + return torch.cat(loss_chunks).sum() / non_masked_elems.maximum(torch.ones_like(non_masked_elems)) def map_old_state_dict_weights(state_dict: Dict, mapping: Mapping, prefix: str) -> Dict: @@ -385,7 +397,7 @@ def __iter__(self) -> Self: def copy_config_files(source_dir: Path, out_dir: Path) -> None: """Copies the specified configuration and tokenizer files into the output directory.""" - config_files = ["generation_config.json", "model_config.yaml"] + config_files = ["config.json", "generation_config.json", "model_config.yaml"] tokenizer_files = ["tokenizer.json", "tokenizer.model", "tokenizer_config.json"] for file_name in config_files + tokenizer_files: diff --git a/pyproject.toml b/pyproject.toml index 1d3c89cfd9..ba3bc7c9e9 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "litgpt" -version = "0.3.0.dev0" +version = "0.4.0.dev0" description = "Hackable implementation of state-of-the-art open-source LLMs" authors = [ { name = "Lightning AI", email = "contact@lightning.ai" }, @@ -11,7 +11,7 @@ license = { file = "LICENSE" } dependencies = [ "torch>=2.2.0", "lightning==2.3.0.dev20240328", - "jsonargparse[signatures]>=4.27.6", + "jsonargparse[signatures]>=4.27.6" ] [project.urls] @@ -37,6 +37,7 @@ all = [ "tokenizers>=0.15.2", # pythia, falcon, redpajama "requests>=2.31.0", # litgpt.data "litdata>=0.2.2", # litgpt.data + "litserve>=0.1.0", # litgpt.deploy "zstandard>=0.22.0", # litgpt.data.prepare_slimpajama.py "pandas>=1.9.0", # litgpt.data.prepare_starcoder.py "pyarrow>=15.0.2", # litgpt.data.prepare_starcoder.py diff --git a/tests/test_adapter.py b/tests/test_adapter.py index cb9ac7b019..2028a78b83 100644 --- a/tests/test_adapter.py +++ b/tests/test_adapter.py @@ -98,7 +98,8 @@ def test_adapter_script(tmp_path, fake_checkpoint_dir, monkeypatch, alpaca_path) logs = stdout.getvalue() assert logs.count("(step)") == 6 - assert logs.count("val loss") == 3 + assert logs.count("val loss") == 4 # 3 validations + 1 final validation + assert logs.count("Final evaluation") == 1 assert "of trainable parameters: 168" in logs diff --git a/tests/test_adapter_v2.py b/tests/test_adapter_v2.py index 67f0689c05..33f00a3166 100644 --- a/tests/test_adapter_v2.py +++ b/tests/test_adapter_v2.py @@ -115,7 +115,8 @@ def test_adapter_v2_script(tmp_path, fake_checkpoint_dir, monkeypatch, alpaca_pa logs = stdout.getvalue() assert logs.count("(step)") == 6 - assert logs.count("val loss") == 3 + assert logs.count("val loss") == 4 # 3 validations + 1 final validation + assert logs.count("Final evaluation") == 1 assert "of trainable parameters: 552" in logs diff --git a/tests/test_cli.py b/tests/test_cli.py index 2c994fcf96..f95841ddc0 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -15,7 +15,7 @@ def test_cli(): main() out = out.getvalue() assert "usage: litgpt" in out - assert "{download,chat,finetune,pretrain,generate,convert,merge_lora,evaluate}" in out + assert "{download,chat,finetune,pretrain,generate,convert,merge_lora,evaluate,serve}" in out assert ( """Available subcommands: download Download weights or tokenizer data from the Hugging @@ -23,19 +23,8 @@ def test_cli(): chat Chat with a model.""" in out ) - assert ("""evaluate Evaluate a model with the LM Evaluation Harness.""") in out - - out = StringIO() - with pytest.raises(SystemExit), redirect_stdout(out), mock.patch("sys.argv", ["litgpt", "finetune", "-h"]): - main() - out = out.getvalue() - assert ( - """Available subcommands: - lora Finetune a model with LoRA. - full Finetune a model.""" - in out - ) - + assert """evaluate Evaluate a model with the LM Evaluation Harness.""" in out + assert """serve Serve and deploy a model with LitServe.""" in out out = StringIO() with pytest.raises(SystemExit), redirect_stdout(out), mock.patch("sys.argv", ["litgpt", "finetune", "lora", "-h"]): main() @@ -61,3 +50,13 @@ def test_cli(): Optional[int], default: 3000000000000)""" in out ) + + +def test_rewrite_finetune_command(): + out1 = StringIO() + with pytest.raises(SystemExit), redirect_stdout(out1), mock.patch("sys.argv", ["litgpt", "fineune", "-h"]): + main() + out2 = StringIO() + with pytest.raises(SystemExit), redirect_stdout(out2), mock.patch("sys.argv", ["litgpt", "fineune", "lora", "-h"]): + main() + assert out1.getvalue() == out2.getvalue() diff --git a/tests/test_convert_lit_checkpoint.py b/tests/test_convert_lit_checkpoint.py index f44609a4f1..ca4ee9881e 100644 --- a/tests/test_convert_lit_checkpoint.py +++ b/tests/test_convert_lit_checkpoint.py @@ -222,73 +222,13 @@ def test_against_original_open_llama_3b(): @torch.inference_mode() -def test_against_hf_phi_1_5(): - wd = Path(__file__).parent.parent.absolute() - workdir = wd / "tests" / "reference_models" - workdir.mkdir(parents=True, exist_ok=True) - file_paths = [workdir / "original_phi_1_5.py", workdir / "configuration_phi.py"] - urls = [ - "https://huggingface.co/microsoft/phi-1_5/raw/main/modeling_phi.py", - "https://huggingface.co/microsoft/phi-1_5/raw/main/configuration_phi.py", - ] - for file_path, url in zip(file_paths, urls): - if not file_path.is_file(): - urlretrieve(url=url, filename=file_path) - - from reference_models.configuration_phi import PhiConfig - from reference_models.original_phi_1_5 import PhiForCausalLM +@pytest.mark.parametrize("model_name", ("phi-1_5", "phi-2")) +def test_against_hf_phi(model_name): + from transformers.models.phi.configuration_phi import PhiConfig + from transformers.models.phi.modeling_phi import PhiForCausalLM ours_config = Config.from_name( - "phi-1_5", padded_vocab_size=10000, n_layer=2, n_head=4, n_embd=256, rotary_percentage=0.5 - ) - T = 5 - theirs_config = PhiConfig( - vocab_size=ours_config.padded_vocab_size, - max_position_embeddings=ours_config.block_size, - hidden_size=ours_config.n_embd, - intermediate_size=ours_config.intermediate_size, - num_attention_heads=ours_config.n_head, - num_hidden_layers=ours_config.n_layer, - partial_rotary_factor=ours_config.rotary_percentage, - ) - - ours_model = GPT(ours_config) - ours_state_dict = ours_model.state_dict() - theirs_state_dict = {} - copy_weights_phi(ours_config, theirs_state_dict, ours_state_dict) - theirs_model = PhiForCausalLM(theirs_config) - # strict=False because we don't save the rotary embeddings inv frequency - keys = theirs_model.load_state_dict(theirs_state_dict, strict=False) - assert not keys.unexpected_keys - assert all("inv_freq" in k for k in keys.missing_keys) - - # test end to end - x = torch.tensor([[9856, 23, 491, 1536, 304]], dtype=torch.int32) - assert x.size(1) == T - ours_y = ours_model(x) - theirs_y = theirs_model(x)["logits"] - torch.testing.assert_close(ours_y, theirs_y) - - -@torch.inference_mode() -def test_against_hf_phi_2(): - wd = Path(__file__).parent.parent.absolute() - workdir = wd / "tests" / "reference_models" - workdir.mkdir(parents=True, exist_ok=True) - file_paths = [workdir / "original_phi_2.py", workdir / "configuration_phi.py"] - urls = [ - "https://huggingface.co/microsoft/phi-2/raw/main/modeling_phi.py", - "https://huggingface.co/microsoft/phi-2/raw/main/configuration_phi.py", - ] - for file_path, url in zip(file_paths, urls): - if not file_path.is_file(): - urlretrieve(url=url, filename=file_path) - - from reference_models.configuration_phi import PhiConfig - from reference_models.original_phi_2 import PhiForCausalLM - - ours_config = Config.from_name( - "phi-2", padded_vocab_size=10000, n_layer=2, n_head=4, n_embd=256, rotary_percentage=0.5 + model_name, padded_vocab_size=10000, n_layer=2, n_head=4, n_embd=256, rotary_percentage=0.5 ) T = 5 theirs_config = PhiConfig( diff --git a/tests/test_evaluate.py b/tests/test_evaluate.py index 023621db6a..12f8a68f9c 100644 --- a/tests/test_evaluate.py +++ b/tests/test_evaluate.py @@ -1,6 +1,5 @@ # Copyright Lightning AI. Licensed under the Apache License 2.0, see LICENSE file. -import shutil import subprocess import sys from contextlib import redirect_stdout @@ -9,7 +8,6 @@ from pathlib import Path from unittest import mock -import datasets import pytest import torch import yaml @@ -19,37 +17,30 @@ from litgpt.scripts.download import download_from_hub -@pytest.mark.xfail( - raises=(datasets.builder.DatasetGenerationError, NotImplementedError), - strict=False, - match="Loading a dataset cached in a LocalFileSystem is not supported", -) -def test_evaluate_script(tmp_path, monkeypatch): +def test_evaluate_script(tmp_path): ours_config = Config.from_name("pythia-14m") download_from_hub(repo_id="EleutherAI/pythia-14m", tokenizer_only=True, checkpoint_dir=tmp_path) - shutil.move(str(tmp_path / "EleutherAI" / "pythia-14m" / "tokenizer.json"), str(tmp_path)) - shutil.move(str(tmp_path / "EleutherAI" / "pythia-14m" / "tokenizer_config.json"), str(tmp_path)) + checkpoint_dir = tmp_path / "EleutherAI" / "pythia-14m" ours_model = GPT(ours_config) - checkpoint_path = tmp_path / "lit_model.pth" - torch.save(ours_model.state_dict(), checkpoint_path) - config_path = tmp_path / "model_config.yaml" - with open(config_path, "w", encoding="utf-8") as fp: + torch.save(ours_model.state_dict(), checkpoint_dir / "lit_model.pth") + with open( checkpoint_dir / "model_config.yaml", "w", encoding="utf-8") as fp: yaml.dump(asdict(ours_config), fp) - fn_kwargs = dict( - checkpoint_dir=tmp_path, - out_dir=tmp_path / "out_dir", - device="cpu", - dtype=torch.float32, - limit=5, - tasks="mathqa" - ) stdout = StringIO() with redirect_stdout(stdout), mock.patch("sys.argv", ["eval/evaluate.py"]): - module.convert_and_evaluate(**fn_kwargs) + module.convert_and_evaluate( + checkpoint_dir=checkpoint_dir, + out_dir=tmp_path / "out_dir", + device=None, + dtype=torch.float32, + limit=5, + tasks="mathqa" + ) stdout = stdout.getvalue() + assert (tmp_path / "out_dir" / "results.json").is_file() assert "mathqa" in stdout assert "Metric" in stdout + assert "Loading checkpoint shards" not in stdout @pytest.mark.parametrize("mode", ["file", "entrypoint"]) diff --git a/tests/test_full.py b/tests/test_full.py index b1ddec5455..74bc10f22e 100644 --- a/tests/test_full.py +++ b/tests/test_full.py @@ -55,7 +55,8 @@ def test_full_script(tmp_path, fake_checkpoint_dir, monkeypatch, alpaca_path): logs = stdout.getvalue() assert logs.count("(step)") == 6 - assert logs.count("val loss") == 3 + assert logs.count("val loss") == 4 # 3 validations + 1 final validation + assert logs.count("Final evaluation") == 1 assert "of trainable parameters: 1,888" in logs # Resume training and do 2 steps more diff --git a/tests/test_lora.py b/tests/test_lora.py index 3a6eeb8de3..c09d07ee66 100644 --- a/tests/test_lora.py +++ b/tests/test_lora.py @@ -107,7 +107,7 @@ def test_lora_mqa_gqa(): assert attn.linear.weight.shape == (24, 8) assert attn.lora_A.shape == (4, 8) assert attn.lora_B.shape == (16, 2) - assert attn.lora_ind == lora_ind + torch.testing.assert_allclose(attn._lora_ind, torch.tensor(lora_ind)) x = torch.randint(0, 8, size=(3, 5, 16), dtype=torch.int64) assert attn.zero_pad(x).shape == (3, 5, 24) bsz, ctx_len, in_dim = 2, 30, 8 @@ -128,7 +128,7 @@ def test_lora_mqa_gqa(): assert attn.linear.weight.shape == (12, 8) assert attn.lora_A.shape == (4, 8) assert attn.lora_B.shape == (10, 2) - assert attn.lora_ind == lora_ind + torch.testing.assert_allclose(attn._lora_ind, torch.tensor(lora_ind)) x = torch.randint(0, 8, size=(3, 5, 10), dtype=torch.int64) assert attn.zero_pad(x).shape == (3, 5, 12) bsz, ctx_len, in_dim = 2, 30, 8 @@ -149,7 +149,7 @@ def test_lora_mqa_gqa(): assert attn.linear.weight.shape == (16, 8) assert attn.lora_A.shape == (4, 8) assert attn.lora_B.shape == (12, 2) - assert attn.lora_ind == lora_ind + torch.testing.assert_allclose(attn._lora_ind, torch.tensor(lora_ind)) x = torch.randint(0, 8, size=(3, 5, 12), dtype=torch.int64) assert attn.zero_pad(x).shape == (3, 5, 16) bsz, ctx_len, in_dim = 2, 30, 8 @@ -221,7 +221,8 @@ def test_lora_script(tmp_path, fake_checkpoint_dir, monkeypatch, alpaca_path): logs = stdout.getvalue() assert logs.count("(step)") == 6 - assert logs.count("val loss") == 3 + assert logs.count("val loss") == 4 # 3 validations + 1 final validation + assert logs.count("Final evaluation") == 1 assert "of trainable parameters: 512" in logs diff --git a/tests/test_model.py b/tests/test_model.py index 7bc0ccb5b4..7743c4f143 100644 --- a/tests/test_model.py +++ b/tests/test_model.py @@ -206,7 +206,13 @@ def test_against_original_open_llama_3b(device, dtype): @torch.inference_mode() @pytest.mark.parametrize( "ours_kwargs", - [{"name": "Llama-2-7b-hf"}, {"name": "CodeLlama-7b-hf"}, {"name": "Llama-2-70b-chat-hf", "n_query_groups": 1}], + [ + {"name": "Llama-2-7b-hf"}, + {"name": "CodeLlama-7b-hf"}, + {"name": "Llama-2-70b-chat-hf", "n_query_groups": 1}, + {"name": "Llama-3-8B"}, + {"name": "Llama-3-8B-Instruct"}, + ], ) @pytest.mark.parametrize( ("device", "dtype"), @@ -224,7 +230,7 @@ def test_against_original_open_llama_3b(device, dtype): ), ], ) -def test_against_hf_llama2(ours_kwargs, device, dtype): +def test_against_hf_llama_2_and_3(ours_kwargs, device, dtype): torch.set_default_dtype(dtype) ours_config = Config.from_name( @@ -261,6 +267,7 @@ def test_against_hf_llama2(ours_kwargs, device, dtype): @torch.inference_mode() +@pytest.mark.parametrize("model_name", ("phi-1_5", "phi-2")) @pytest.mark.parametrize( ("device", "dtype"), [ @@ -272,86 +279,14 @@ def test_against_hf_llama2(ours_kwargs, device, dtype): ), ], ) -def test_against_hf_phi_1_5(device, dtype): - wd = Path(__file__).parent.parent.resolve() - workdir = wd / "tests" / "reference_models" - workdir.mkdir(parents=True, exist_ok=True) - file_paths = [workdir / "original_phi_1_5.py", workdir / "configuration_phi.py"] - urls = [ - "https://huggingface.co/microsoft/phi-1_5/raw/main/modeling_phi.py", - "https://huggingface.co/microsoft/phi-1_5/raw/main/configuration_phi.py", - ] - for file_path, url in zip(file_paths, urls): - if not file_path.is_file(): - urlretrieve(url=url, filename=file_path) - - from reference_models.configuration_phi import PhiConfig - from reference_models.original_phi_1_5 import PhiForCausalLM - - torch.set_default_dtype(dtype) - - ours_config = Config.from_name( - "phi-1_5", padded_vocab_size=10000, n_layer=2, n_head=4, n_embd=256, rotary_percentage=0.5 - ) - T = 5 - theirs_config = PhiConfig( - vocab_size=ours_config.padded_vocab_size, - max_position_embeddings=ours_config.block_size, - hidden_size=ours_config.n_embd, - intermediate_size=ours_config.intermediate_size, - num_attention_heads=ours_config.n_head, - num_hidden_layers=ours_config.n_layer, - partial_rotary_factor=ours_config.rotary_percentage, - torch_dtype=dtype, - ) - - theirs_model = PhiForCausalLM(theirs_config).to(device) - theirs_state_dict = theirs_model.state_dict() - state_dict = {} - copy_weights_phi(ours_config, {}, state_dict, theirs_state_dict) - ours_model = GPT(ours_config).to(device) - ours_model.load_state_dict(state_dict) - - # test end to end - x = torch.tensor([[9856, 23, 491, 1536, 304]], dtype=torch.int32, device=device) - assert x.size(1) == T - ours_y = ours_model(x) - theirs_y = theirs_model(x)["logits"].to(dtype) # HF converts logits to float - torch.testing.assert_close(ours_y, theirs_y) - - -@torch.inference_mode() -@pytest.mark.parametrize( - ("device", "dtype"), - [ - (torch.device("cpu"), torch.float32), - pytest.param( - torch.device("cuda"), - torch.float16, - marks=[pytest.mark.xfail(raises=AssertionError, strict=False), RunIf(min_cuda_gpus=1)], - ), - ], -) -def test_against_hf_phi_2(device, dtype): - wd = Path(__file__).parent.parent.resolve() - workdir = wd / "tests" / "reference_models" - workdir.mkdir(parents=True, exist_ok=True) - file_paths = [workdir / "original_phi_2.py", workdir / "configuration_phi.py"] - urls = [ - "https://huggingface.co/microsoft/phi-2/raw/main/modeling_phi.py", - "https://huggingface.co/microsoft/phi-2/raw/main/configuration_phi.py", - ] - for file_path, url in zip(file_paths, urls): - if not file_path.is_file(): - urlretrieve(url=url, filename=file_path) - - from reference_models.configuration_phi import PhiConfig - from reference_models.original_phi_2 import PhiForCausalLM +def test_against_hf_phi(model_name, device, dtype): + from transformers.models.phi.configuration_phi import PhiConfig + from transformers.models.phi.modeling_phi import PhiForCausalLM torch.set_default_dtype(dtype) ours_config = Config.from_name( - "phi-2", padded_vocab_size=10000, n_layer=2, n_head=4, n_embd=256, rotary_percentage=0.5 + model_name, padded_vocab_size=10000, n_layer=2, n_head=4, n_embd=256, rotary_percentage=0.5 ) T = 5 theirs_config = PhiConfig( diff --git a/tests/test_pretrain.py b/tests/test_pretrain.py index 61c67608af..d252524e87 100644 --- a/tests/test_pretrain.py +++ b/tests/test_pretrain.py @@ -13,10 +13,11 @@ from lightning.fabric.strategies import FSDPStrategy, SingleDeviceStrategy from torch.utils.data import DataLoader +from test_utils import test_init_out_dir from litgpt import pretrain from litgpt.args import EvalArgs, TrainArgs from litgpt.config import Config -from litgpt.pretrain import init_out_dir, initialize_weights +from litgpt.pretrain import initialize_weights @RunIf(min_cuda_gpus=2, standalone=True) @@ -89,17 +90,6 @@ def test_pretrain_model_name_and_config(): pretrain.setup(model_name="tiny-llama-1.1b", model_config=Config(name="tiny-llama-1.1b")) -def test_init_out_dir(tmp_path): - relative_path = Path("./out") - absolute_path = tmp_path / "out" - assert init_out_dir(relative_path) == relative_path - assert init_out_dir(absolute_path) == absolute_path - - with mock.patch.dict(os.environ, {"LIGHTNING_ARTIFACTS_DIR": "prefix"}): - assert init_out_dir(relative_path) == Path("prefix") / relative_path - assert init_out_dir(absolute_path) == absolute_path - - @pytest.mark.parametrize(("strategy", "expected"), [(SingleDeviceStrategy, True), (FSDPStrategy, False)]) def test_initialize_weights(strategy, expected): fabric_mock = Mock() diff --git a/tests/test_prompts.py b/tests/test_prompts.py index 3250ce4801..20f2c84e0c 100644 --- a/tests/test_prompts.py +++ b/tests/test_prompts.py @@ -50,6 +50,8 @@ def test_prompt_style_from_config(): "Llama-2-7b-chat-hf", "Llama-2-13b-chat-hf", "Llama-2-70b-chat-hf", + "Llama-3-8B-Instruct", + "Llama-3-70B-Instruct", "Gemma-2b-it", "Gemma-7b-it", "FreeWilly2", diff --git a/tests/test_serve.py b/tests/test_serve.py new file mode 100644 index 0000000000..46a109c807 --- /dev/null +++ b/tests/test_serve.py @@ -0,0 +1,42 @@ +# Copyright Lightning AI. Licensed under the Apache License 2.0, see LICENSE file. +from dataclasses import asdict +import shutil + +from lightning.fabric import seed_everything +from fastapi.testclient import TestClient +from litserve.server import LitServer +import torch +import yaml + + +from litgpt import GPT, Config +from litgpt.deploy.serve import SimpleLitAPI +from litgpt.scripts.download import download_from_hub + + +def test_simple(tmp_path): + + # Create model checkpoint + seed_everything(123) + ours_config = Config.from_name("pythia-14m") + download_from_hub(repo_id="EleutherAI/pythia-14m", tokenizer_only=True, checkpoint_dir=tmp_path) + shutil.move(str(tmp_path / "EleutherAI" / "pythia-14m" / "tokenizer.json"), str(tmp_path)) + shutil.move(str(tmp_path / "EleutherAI" / "pythia-14m" / "tokenizer_config.json"), str(tmp_path)) + ours_model = GPT(ours_config) + checkpoint_path = tmp_path / "lit_model.pth" + torch.save(ours_model.state_dict(), checkpoint_path) + config_path = tmp_path / "model_config.yaml" + with open(config_path, "w", encoding="utf-8") as fp: + yaml.dump(asdict(ours_config), fp) + + accelerator = "cpu" + server = LitServer( + SimpleLitAPI(checkpoint_dir=tmp_path, temperature=1, top_k=1), + accelerator=accelerator, devices=1, timeout=60 + ) + + with TestClient(server.app) as client: + response = client.post("/predict", json={"prompt": "Hello world"}) + # Model is a small random model, not trained, hence the gibberish. + # We are just testing that the server works. + assert response.json()["output"][:19] == "Hello world statues" diff --git a/tests/test_tokenizer.py b/tests/test_tokenizer.py index f9aede3921..b7e4d0b33f 100644 --- a/tests/test_tokenizer.py +++ b/tests/test_tokenizer.py @@ -11,6 +11,7 @@ from litgpt.tokenizer import Tokenizer +@pytest.mark.flaky(reruns=5) @pytest.mark.parametrize("config", config_module.configs, ids=[c["hf_config"]["name"] for c in config_module.configs]) def test_tokenizer_against_hf(config): access_token = os.getenv("HF_TOKEN") diff --git a/tests/test_utils.py b/tests/test_utils.py index 99d883a3f2..cbb5230621 100644 --- a/tests/test_utils.py +++ b/tests/test_utils.py @@ -30,6 +30,7 @@ copy_config_files, find_multiple, incremental_save, + init_out_dir, num_parameters, parse_devices, save_hyperparameters, @@ -294,3 +295,14 @@ def test_choose_logger(tmp_path): with pytest.raises(ValueError, match="`--logger_name=foo` is not a valid option."): choose_logger("foo", out_dir=tmp_path, name="foo") + + +def test_init_out_dir(tmp_path): + relative_path = Path("./out") + absolute_path = tmp_path / "out" + assert init_out_dir(relative_path) == relative_path + assert init_out_dir(absolute_path) == absolute_path + + with mock.patch.dict(os.environ, {"LIGHTNING_ARTIFACTS_DIR": "prefix"}): + assert init_out_dir(relative_path) == Path("prefix") / relative_path + assert init_out_dir(absolute_path) == absolute_path \ No newline at end of file diff --git a/tutorials/0_to_litgpt.md b/tutorials/0_to_litgpt.md index 337bf37049..e5e1c7c765 100644 --- a/tutorials/0_to_litgpt.md +++ b/tutorials/0_to_litgpt.md @@ -464,6 +464,44 @@ litgpt evaluate \ (A list of supported tasks can be found [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/master/docs/task_table.md).) +  +## Deploy LLMs + +You can deploy LitGPT LLMs using your tool of choice. Below is an example using LitGPT built-in serving capabilities: + + +```bash +# 1) Download a pretrained model (alternatively, use your own finetuned model) +litgpt download --repo_id microsoft/phi-2 + +# 2) Start the server +litgpt serve --checkpoint_dir checkpoints/microsoft/phi-2 +``` + +```python +# 3) Use the server (in a separate session) +import requests, json + response = requests.post( + "http://127.0.0.1:8000/predict", + json={"prompt": "Fix typos in the following sentence: Exampel input"} +) +print(response.json()["output"]) +``` + +This prints: + +``` +Instruct: Fix typos in the following sentence: Exampel input +Output: Example input. +``` + + +  +**More information and additional resources** + +- [tutorials/deploy](deploy.md): A full deployment tutorial and example + +   ## Converting LitGPT model weights to `safetensors` format diff --git a/tutorials/deploy.md b/tutorials/deploy.md new file mode 100644 index 0000000000..1b1495fde7 --- /dev/null +++ b/tutorials/deploy.md @@ -0,0 +1,49 @@ +# Serve and Deploy LLMs + +This document shows how you can serve a LitGPT for deployment. + +  +## Serve an LLM + +This section illustrates how we can set up an inference server for a phi-2 LLM using `litgpt serve` that is minimal and highly scalable. + + +  +## Step 1: Start the inference server + + +```bash +# 1) Download a pretrained model (alternatively, use your own finetuned model) +litgpt download --repo_id microsoft/phi-2 + +# 2) Start the server +litgpt serve --checkpoint_dir checkpoints/microsoft/phi-2 +``` + +> [!TIP] +> Use `litgpt serve --help` to display additional options, including the port, devices, LLM temperature setting, and more. + + +  +## Step 2: Query the inference server + +You can now send requests to the inference server you started in step 2. For example, in a new Python session, we can send requests to the inference server as follows: + + +```python +import requests, json + +response = requests.post( + "http://127.0.0.1:8000/predict", + json={"prompt": "Fix typos in the following sentence: Exampel input"} +) + +print(response.json()["output"]) +``` + +Executing the code above prints the following output: + +``` +Instruct: Fix typos in the following sentence: Exampel input +Output: Example input. +``` diff --git a/tutorials/download_model_weights.md b/tutorials/download_model_weights.md index d1c320ac33..45c9c7d50c 100644 --- a/tutorials/download_model_weights.md +++ b/tutorials/download_model_weights.md @@ -3,29 +3,30 @@ LitGPT supports a variety of LLM architectures with publicly available weights. You can download model weights and access a list of supported models using the LitGPT `download.py` script. -| Model | Model size | Reference | -|----------------------------------------------|------------------------------------------|------------------------------------------------------------------------------------------------------------------------------| -| CodeGemma by Google | 7B | [Google Team, Google Deepmind](https://ai.google.dev/gemma/docs/codegemma) | -| Code Llama by Meta AI | 7B, 13B, 34B, 70B | [Rozière et al. 2023](https://arxiv.org/abs/2308.12950) | -| Dolly by Databricks | 3B, 7B, 12B | [Conover et al. 2023](https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm) | -| Falcon by TII UAE | 7B, 40B, 180B | [TII 2023](https://falconllm.tii.ae) | -| FreeWilly2 (Stable Beluga 2) by Stability AI | 70B | [Stability AI 2023](https://stability.ai/blog/stable-beluga-large-instruction-fine-tuned-models) | -| Function Calling Llama 2 by Trelis | 7B | [Trelis et al. 2023](https://huggingface.co/Trelis/Llama-2-7b-chat-hf-function-calling-v2) | -| Gemma by Google | 2B, 7B | [Google Team, Google Deepmind](https://storage.googleapis.com/deepmind-media/gemma/gemma-report.pdf) | -| Llama 2 by Meta AI | 7B, 13B, 70B | [Touvron et al. 2023](https://arxiv.org/abs/2307.09288) | -| LongChat by LMSYS | 7B, 13B | [LongChat Team 2023](https://lmsys.org/blog/2023-06-29-longchat/) | -| Mistral and Mixtral by Mistral AI | 7B | [Mistral website](https://mistral.ai/) | -| Nous-Hermes by NousResearch | 7B, 13B, 70B | [Org page](https://huggingface.co/NousResearch) | -| OpenLLaMA by OpenLM Research | 3B, 7B, 13B | [Geng & Liu 2023](https://github.com/openlm-research/open_llama) | -| Phi by Microsoft Research | 1.3B, 2.7B | [Li et al. 2023](https://arxiv.org/abs/2309.05463) | -| Platypus by Lee at el. | 7B, 13B, 70B | [Lee, Hunter, and Ruiz 2023](https://arxiv.org/abs/2308.07317) | -| Pythia by EleutherAI | {14,31,70,160,410}M, {1,1.4,2.8,6.9,12}B | [Biderman et al. 2023](https://arxiv.org/abs/2304.01373) | -| RedPajama-INCITE by Together | 3B, 7B | [Together 2023](https://together.ai/blog/redpajama-models-v1) | -| StableCode by Stability AI | 3B | [Stability AI 2023](https://stability.ai/blog/stablecode-llm-generative-ai-coding) | -| StableLM by Stability AI | 3B, 7B | [Stability AI 2023](https://github.com/Stability-AI/StableLM) | -| StableLM Zephyr by Stability AI | 3B | [Stability AI 2023](https://stability.ai/blog/stablecode-llm-generative-ai-coding) | -| TinyLlama by Zhang et al. | 1.1B | [Zhang et al. 2023](https://github.com/jzhang38/TinyLlama) | -| Vicuna by LMSYS | 7B, 13B, 33B | [Li et al. 2023](https://lmsys.org/blog/2023-03-30-vicuna/) | +| Model | Model size | Reference | +|----------------------------------------------|-----------------------------------------|--------------------------------------------------------------------------------------------------------------------------| +| CodeGemma by Google | 7B | [Google Team, Google Deepmind](https://ai.google.dev/gemma/docs/codegemma) | +| Code Llama by Meta AI | 7B, 13B, 34B, 70B | [Rozière et al. 2023](https://arxiv.org/abs/2308.12950) | +| Dolly by Databricks | 3B, 7B, 12B | [Conover et al. 2023](https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm) | +| Falcon by TII UAE | 7B, 40B, 180B | [TII 2023](https://falconllm.tii.ae) | +| FreeWilly2 (Stable Beluga 2) by Stability AI | 70B | [Stability AI 2023](https://stability.ai/blog/stable-beluga-large-instruction-fine-tuned-models) | +| Function Calling Llama 2 by Trelis | 7B | [Trelis et al. 2023](https://huggingface.co/Trelis/Llama-2-7b-chat-hf-function-calling-v2) | +| Gemma by Google | 2B, 7B | [Google Team, Google Deepmind](https://storage.googleapis.com/deepmind-media/gemma/gemma-report.pdf) | +| Llama 2 by Meta AI | 7B, 13B, 70B | [Touvron et al. 2023](https://arxiv.org/abs/2307.09288) | +| Llama 3 by Meta AI | 8B, 70B | [Meta AI 2024](https://github.com/meta-llama/llama3) | +| LongChat by LMSYS | 7B, 13B | [LongChat Team 2023](https://lmsys.org/blog/2023-06-29-longchat/) | +| Mistral and Mixtral by Mistral AI | 7B | [Mistral website](https://mistral.ai/) | +| Nous-Hermes by NousResearch | 7B, 13B, 70B | [Org page](https://huggingface.co/NousResearch) | +| OpenLLaMA by OpenLM Research | 3B, 7B, 13B | [Geng & Liu 2023](https://github.com/openlm-research/open_llama) | +| Phi by Microsoft Research | 1.3B, 2.7B | [Li et al. 2023](https://arxiv.org/abs/2309.05463) | +| Platypus by Lee at el. | 7B, 13B, 70B | [Lee, Hunter, and Ruiz 2023](https://arxiv.org/abs/2308.07317) | +| Pythia by EleutherAI | {14,31,70,160,410}M, {1,1.4,2.8,6.9,12}B | [Biderman et al. 2023](https://arxiv.org/abs/2304.01373) | +| RedPajama-INCITE by Together | 3B, 7B | [Together 2023](https://together.ai/blog/redpajama-models-v1) | +| StableCode by Stability AI | 3B | [Stability AI 2023](https://stability.ai/blog/stablecode-llm-generative-ai-coding) | +| StableLM by Stability AI | 3B, 7B | [Stability AI 2023](https://github.com/Stability-AI/StableLM) | +| StableLM Zephyr by Stability AI | 3B | [Stability AI 2023](https://stability.ai/blog/stablecode-llm-generative-ai-coding) | +| TinyLlama by Zhang et al. | 1.1B | [Zhang et al. 2023](https://github.com/jzhang38/TinyLlama) | +| Vicuna by LMSYS | 7B, 13B, 33B | [Li et al. 2023](https://lmsys.org/blog/2023-03-30-vicuna/) | @@ -105,6 +106,10 @@ meta-llama/Llama-2-70b-chat-hf meta-llama/Llama-2-70b-hf meta-llama/Llama-2-7b-chat-hf meta-llama/Llama-2-7b-hf +meta-llama/Meta-Llama-3-70B +meta-llama/Meta-Llama-3-70B-Instruct +meta-llama/Meta-Llama-3-8B +meta-llama/Meta-Llama-3-8B-Instruct microsoft/phi-1_5 microsoft/phi-2 mistralai/Mistral-7B-Instruct-v0.1