📝 Better leaderboards

FleksySDK · May 28, 2024 · 94e7326 · 94e7326
1 parent a44c2d4
commit 94e7326
Show file tree

Hide file tree

Showing 3 changed files with 47 additions and 13 deletions.
diff --git a/docs/leaderboards/compare.md b/docs/leaderboards/compare.md
@@ -9,16 +9,45 @@ hide:
 [//]: # (There is a Mkdocs hook (defined in `docs/hooks.py`) that will read the content of this page. Any line starting with `>>>` will be extracted and replaced with the scores found in the corresponding result file.)
 [//]: # (The format to follow is : `>>>{name}|{result_file_name}|{optional_additional_fields}`)
 
-| Keyboard | Score | Typo detection rate | Auto-correction frustration rate | Auto-completion | Next-word prediction | SDK available |
-|---------:|:-----:|:-------------------:|:--------------------------------:|:---------------:|:-------------------:|:-------------:|
+| Keyboard | Overall score | Typo detection rate | Auto-correction frustration rate | Auto-completion success rate | Next-word prediction success rate | SDK available |
+|---------:|:-------------:|:-------------------:|:--------------------------------:|:---------------:|:-------------------:|:-------------:|
 >>>Fleksy|results/fleksy.json|:fontawesome-solid-circle-check:{ .v_icon }
 >>>iOS keyboard|results/ios.json|:fontawesome-regular-circle-xmark:{ .x_icon }
 >>>KeyboardKit Open-source|results/keyboardkit_oss.json|:fontawesome-solid-circle-check:{ .v_icon }
 >>>KeyboardKit Pro|results/keyboardkit_pro.json|:fontawesome-solid-circle-check:{ .v_icon }
 >>>Gboard|results/gboard.json|:fontawesome-regular-circle-xmark:{ .x_icon }
 >>>Swiftkey|results/swiftkey.json|:fontawesome-regular-circle-xmark:{ .x_icon }
->>>Tappa|results/tappa.json|:fontawesome-regular-circle-xmark:{ .x_icon }
+>>>Tappa|results/tappa.json|:fontawesome-solid-circle-check:{ .v_icon }
 >>>Yandex|results/yandex.json|:fontawesome-regular-circle-xmark:{ .x_icon }
 
-!!! info
-    This leaderboard uses the data from the [raw leaderboard](main.md).
+### Metrics
+
+=== "Overall score"
+
+    A single, general score representing the performances of the keyboard across all tasks.
+
+    :material-trending-up: _Higher is better._
+
+=== "Typo detection rate"
+
+    Percentage of typos detected and corrected by the keyboard.
+
+    :material-trending-up: _Higher is better._
+
+=== "Auto-correction frustration rate"
+
+    Percentage of words correctly typed, but corrected to something else by the keyboard.
+
+    :material-trending-down: _Lower is better._
+
+=== "Auto-completion success rate"
+
+    Percentage of words correctly auto-completed.
+
+    :material-trending-up: _Higher is better._
+
+=== "Next-word prediction success rate"
+
+    Percentage of words correctly predicted from the context.
+
+    :material-trending-up: _Higher is better._
diff --git a/docs/leaderboards/main.md b/docs/leaderboards/main.md
@@ -4,8 +4,8 @@
 [//]: # (There is a Mkdocs hook (defined in `docs/hooks.py`) that will read the content of this page. Any line starting with `>>>` will be extracted and replaced with the scores found in the corresponding result file.)
 [//]: # (The format to follow is : `>>>{name}|{result_file_name}|{optional_additional_fields}`)
 
-| Keyboard | Score | Auto-correction | Auto-completion | Next-word prediction |
-|---------:|:-----:|:---------------:|:---------------:|:--------------------:|
+| Keyboard | Overall<br>score | Auto-correction | Auto-completion | Next-word prediction |
+|---------:|:----------------:|:---------------:|:---------------:|:--------------------:|
 >>>Fleksy|results/fleksy.json
 >>>iOS keyboard|results/ios.json
 >>>KeyboardKit Open-source|results/keyboardkit_oss.json
@@ -15,13 +15,15 @@
 >>>Tappa|results/tappa.json
 >>>Yandex|results/yandex.json
 
-!!! info
-    The metrics used in this leaderboard are :
+---
 
-    * For next-word prediction : top-3 accuracy
-    * For auto-completion : top-3 accuracy
-    * For auto-correction : F-score
+The metrics used in this leaderboard are :
 
+* Auto-correction : _**F-score**_
+* Auto-completion : _**top-3 accuracy**_
+* Next-word prediction : _**top-3 accuracy**_
+
+!!! tip
     See [Understanding the metrics](../how_testing_is_done.md#understanding-the-metrics) for more details.
 
-    The overall score is a _weighted sum_ of each task's score.
+The overall score is a _**weighted sum**_ of all tasks.
diff --git a/mkdocs.yml b/mkdocs.yml
@@ -37,6 +37,9 @@ markdown_extensions:
   - pymdownx.emoji:
       emoji_index: !!python/name:material.extensions.emoji.twemoji
       emoji_generator: !!python/name:material.extensions.emoji.to_svg
+  - pymdownx.superfences
+  - pymdownx.tabbed:
+      alternate_style: true
 
 nav:
   - Welcome: "index.md"