Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Switch to ICU tokenizer #939

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Switch to ICU tokenizer #939

wants to merge 6 commits into from

Commits on Nov 22, 2024

  1. Add ICU tokenizer

    @eu9ene
    eu9ene committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    4041e04 View commit details
    Browse the repository at this point in the history

  2. Use ICU tokenizer in alignments

    @eu9ene
    eu9ene committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    070c0d3 View commit details
    Browse the repository at this point in the history

  3. Update to OpusTrainer with ICU detokenization support

    @eu9ene
    eu9ene committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    4000853 View commit details
    Browse the repository at this point in the history

  4. Update docs

    @eu9ene
    eu9ene committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    13a80bf View commit details
    Browse the repository at this point in the history

  5. Add pyicu pypi package

    @eu9ene
    eu9ene committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    bb7f523 View commit details
    Browse the repository at this point in the history

  6. Use ICU system package

    @eu9ene
    eu9ene committed Nov 22, 2024
    1 Configuration menu
    Copy the full SHA
    d585a63 View commit details
    Browse the repository at this point in the history