Skip to content

Commit

Permalink
Update llama-learns-to-talk.mdx
Browse files Browse the repository at this point in the history
  • Loading branch information
hahuyhoang411 authored Oct 7, 2024
1 parent 1f8d5f8 commit ffb6108
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions src/pages/blog/llama-learns-to-talk.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ Spoiler alert: We recovered MMLU performance from 0.42 to **0.63**, reducing the
| Test 1: Early Pretrain Recovery | 3,000 steps | 500k mixed |||| 0.55 |
| Test 2: Late Pretrain Recovery | Last | 500k mixed |||| 0.515 |
| Test 3: Late Pretrain Recovery with Transcription (With transcription token) | Last | 500k mixed |||| 0.48 |
| Test 4: Extended Late Pretrain Recovery (With transcription prompts) | Last | 1.89M mixed |||| 0.63 |
| Test 4: Extended Late Pretrain Recovery (With transcription prompts) | Last | 1.89M mixed |||| **0.63** |

**Mixed training data between modalities:** We determined an optimal interleaving of different data types with 70% speech instruction prompts, 20% speech transcription prompts and 10% text-only prompts.

Expand Down Expand Up @@ -215,7 +215,7 @@ Beyond randomizing sound tokens for inaudible input, we also performed sequence
| [Llama3.1-s-v2](https://huggingface.co/homebrewltd/llama3-s-instruct-v0.2) | 3.45 | 3.53 |
| [Ichigo-llama3.1-s v0.3-phase2 -cp7000](https://huggingface.co/homebrewltd/Ichigo-llama3.1-s-instruct-v0.3-phase-2) | 3.42 | 3.62 |
| [Ichigo-llama3.1-s v0.3-phase2-cplast](https://huggingface.co/jan-hq/llama3-s-instruct-v0.3-checkpoint-last) | 3.31 | 3.6 |
| [Ichigo-llama3.1-s v0.3-phase3](https://huggingface.co/homebrewltd/Ichigo-llama3.1-s-instruct-v0.3-phase-3) | 3.64 | 3.68 |
| [Ichigo-llama3.1-s v0.3-phase3](https://huggingface.co/homebrewltd/Ichigo-llama3.1-s-instruct-v0.3-phase-3) | **3.64** | **3.68** |
| [Qwen2-audio-7B](https://huggingface.co/Qwen/Qwen2-Audio-7B) | 2.63 | 2.24 |

## Next steps
Expand Down

0 comments on commit ffb6108

Please sign in to comment.