diff --git a/docs/leaderboards/compare.md b/docs/leaderboards/compare.md index e6c3cfd..764706c 100644 --- a/docs/leaderboards/compare.md +++ b/docs/leaderboards/compare.md @@ -9,16 +9,45 @@ hide: [//]: # (There is a Mkdocs hook (defined in `docs/hooks.py`) that will read the content of this page. Any line starting with `>>>` will be extracted and replaced with the scores found in the corresponding result file.) [//]: # (The format to follow is : `>>>{name}|{result_file_name}|{optional_additional_fields}`) -| Keyboard | Score | Typo detection rate | Auto-correction frustration rate | Auto-completion | Next-word prediction | SDK available | -|---------:|:-----:|:-------------------:|:--------------------------------:|:---------------:|:-------------------:|:-------------:| +| Keyboard | Overall score | Typo detection rate | Auto-correction frustration rate | Auto-completion success rate | Next-word prediction success rate | SDK available | +|---------:|:-------------:|:-------------------:|:--------------------------------:|:---------------:|:-------------------:|:-------------:| >>>Fleksy|results/fleksy.json|:fontawesome-solid-circle-check:{ .v_icon } >>>iOS keyboard|results/ios.json|:fontawesome-regular-circle-xmark:{ .x_icon } >>>KeyboardKit Open-source|results/keyboardkit_oss.json|:fontawesome-solid-circle-check:{ .v_icon } >>>KeyboardKit Pro|results/keyboardkit_pro.json|:fontawesome-solid-circle-check:{ .v_icon } >>>Gboard|results/gboard.json|:fontawesome-regular-circle-xmark:{ .x_icon } >>>Swiftkey|results/swiftkey.json|:fontawesome-regular-circle-xmark:{ .x_icon } ->>>Tappa|results/tappa.json|:fontawesome-regular-circle-xmark:{ .x_icon } +>>>Tappa|results/tappa.json|:fontawesome-solid-circle-check:{ .v_icon } >>>Yandex|results/yandex.json|:fontawesome-regular-circle-xmark:{ .x_icon } -!!! info - This leaderboard uses the data from the [raw leaderboard](main.md). +### Metrics + +=== "Overall score" + + A single, general score representing the performances of the keyboard across all tasks. + + :material-trending-up: _Higher is better._ + +=== "Typo detection rate" + + Percentage of typos detected and corrected by the keyboard. + + :material-trending-up: _Higher is better._ + +=== "Auto-correction frustration rate" + + Percentage of words correctly typed, but corrected to something else by the keyboard. + + :material-trending-down: _Lower is better._ + +=== "Auto-completion success rate" + + Percentage of words correctly auto-completed. + + :material-trending-up: _Higher is better._ + +=== "Next-word prediction success rate" + + Percentage of words correctly predicted from the context. + + :material-trending-up: _Higher is better._ diff --git a/docs/leaderboards/main.md b/docs/leaderboards/main.md index 425dfa7..df36ab8 100644 --- a/docs/leaderboards/main.md +++ b/docs/leaderboards/main.md @@ -4,8 +4,8 @@ [//]: # (There is a Mkdocs hook (defined in `docs/hooks.py`) that will read the content of this page. Any line starting with `>>>` will be extracted and replaced with the scores found in the corresponding result file.) [//]: # (The format to follow is : `>>>{name}|{result_file_name}|{optional_additional_fields}`) -| Keyboard | Score | Auto-correction | Auto-completion | Next-word prediction | -|---------:|:-----:|:---------------:|:---------------:|:--------------------:| +| Keyboard | Overall
score | Auto-correction | Auto-completion | Next-word prediction | +|---------:|:----------------:|:---------------:|:---------------:|:--------------------:| >>>Fleksy|results/fleksy.json >>>iOS keyboard|results/ios.json >>>KeyboardKit Open-source|results/keyboardkit_oss.json @@ -15,13 +15,15 @@ >>>Tappa|results/tappa.json >>>Yandex|results/yandex.json -!!! info - The metrics used in this leaderboard are : +--- - * For next-word prediction : top-3 accuracy - * For auto-completion : top-3 accuracy - * For auto-correction : F-score +The metrics used in this leaderboard are : +* Auto-correction : _**F-score**_ +* Auto-completion : _**top-3 accuracy**_ +* Next-word prediction : _**top-3 accuracy**_ + +!!! tip See [Understanding the metrics](../how_testing_is_done.md#understanding-the-metrics) for more details. - The overall score is a _weighted sum_ of each task's score. +The overall score is a _**weighted sum**_ of all tasks. diff --git a/mkdocs.yml b/mkdocs.yml index 6f9afe0..2554c28 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -37,6 +37,9 @@ markdown_extensions: - pymdownx.emoji: emoji_index: !!python/name:material.extensions.emoji.twemoji emoji_generator: !!python/name:material.extensions.emoji.to_svg + - pymdownx.superfences + - pymdownx.tabbed: + alternate_style: true nav: - Welcome: "index.md"