Skip to content

Commit

Permalink
updates readme
Browse files Browse the repository at this point in the history
  • Loading branch information
djliden committed Dec 4, 2023
1 parent 6c1408f commit 9af52c4
Showing 1 changed file with 8 additions and 0 deletions.
8 changes: 8 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,11 @@ This series of notebooks is intended to show how to fine-tune language models, s
Existing examples and learning resources generally do not bridge the practical gap between single-node single-GPU training when all parameters fit in VRAM, and the various forms of distributed training. These examples, when complete, are intended to show how to train smaller models given sufficient compute resources and then scale the models up until we encounter compute and/or memory constraints. We will then introduce various distributed training approaches aimed at overcoming these issues.

This will, hopefully, serve as a practical and conceptual bridge from single-node single-GPU training to distributed training with tools such as deepspeed and FSDP.

## How to use this repository

The examples in this repository are intended to be read sequentially. Later examples build on earlier examples and gradually add scale and complexity.

## Contents
[1. Fine-Tune T5-Small on Single GPU](./1_t5_small_single_gpu/1.%20T5-Small%20on%20Single%20GPU.ipynb)
[2. Fine-Tune GPT-2 on a Single GPU](./2_gpt2_single_gpu/2.%20GPT-2%20on%20a%20single%20GPU.ipynb)

0 comments on commit 9af52c4

Please sign in to comment.