Skip to content

Issues: GEM-benchmark/GEM-metrics

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

21 Open 40 Closed
21 Open 40 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Issues generating Heavy metrics
#103 opened Feb 27, 2023 by SimonvdFliert
Troubles running embedding metrics in Colab (BERTscore, BLEURT and Prism)
#102 opened Sep 19, 2022 by asnota
2
Key hyperparameters to reproduce results？
#96 opened Jul 1, 2022 by VickiCui
3
Support for Multilingual Rouge
#88 opened Mar 18, 2022 by ratishsp
CUDA out of memory when computing bleurt on GEM submission
#87 opened Mar 16, 2022 by lewtun
Support for iBLEU for paraphrasing
#85 opened Mar 14, 2022 by ratishsp
Add support for docker and add all the metrics from repro enhancement New feature or request
#74 opened Dec 16, 2021 by sebastianGehrmann
1
@danieldeutsch
MAUVE metric for open-ended text generation new-metric-heavy Adding a new resource intensive metric
#72 opened Dec 9, 2021 by asnota
1
MATTR metric new-metric-light Adding a lightweight metric
#71 opened Nov 9, 2021 by ndaheim
@ndaheim
L'Ambre metric new-metric Adding a new metric with unknown resource requirements (could be light or heavy)
#66 opened Oct 23, 2021 by sebastianGehrmann
1
Add a unique versioning key to each metric enhancement New feature or request
#59 opened Oct 20, 2021 by sebastianGehrmann
Change caching to hash of input/output/source instead of filepath based key enhancement New feature or request
#58 opened Oct 20, 2021 by sebastianGehrmann
CLIPScore:A Reference-free Evaluation Metric for Image Captioning new-metric-heavy Adding a new resource intensive metric
#46 opened Apr 24, 2021 by kaustubhdhole
Release some starting guide for participants documentation Improvements or additions to documentation
#45 opened Apr 15, 2021 by sebastianGehrmann
1
Type hints enhancement New feature or request
#30 opened Feb 8, 2021 by madaan
Czech restaurant (and possibly DART) BLEU seems suspiciously low bug Something isn't working
#27 opened Feb 3, 2021 by tuetschek
@tuetschek
14
Additional diversity metrics new-metric-light Adding a lightweight metric
#24 opened Feb 2, 2021 by evanmiltenburg
2
Dataset-specific metrics enhancement New feature or request
#21 opened Feb 1, 2021 by evanmiltenburg
4
Add verbose mode enhancement New feature or request
#19 opened Feb 1, 2021 by evanmiltenburg
Add PARENT new-metric-light Adding a lightweight metric
#14 opened Jan 30, 2021 by sebastianGehrmann
4
Allow multiple different input formats enhancement New feature or request
#2 opened Jan 12, 2021 by tuetschek
1 of 3 tasks
1
ProTip! Exclude everything labeled bug with -label:bug.