Skip to content

Add ignore_merges option to BPE tokenizers#716

Merged
xenova merged 1 commit intomainfrom tokenizers-ignore-mergesApr 18, 2024

Commits