MERA

MERA (Multimodal Evaluation for Russian-language Architectures) is a new open benchmark for the Russian language for evaluating fundamental models.

About MERA

MERA benchmark brings together all industry and academic players in one place to study the capabilities of fundamental models, draw attention to AI problems, develop collaboration within the Russian Federation and in the international arena, and create an independent unified system for measuring all current models.

Our contributions of this project are:

• Instruction-based tasks available in 🤗HuggingFace dataset card link.
• LM-Harness evaluation code for models.
• Website of the benchmark with the Leaderboard and the scoring system inside.
• Baselines of the open models and Human Benchmark.

v1.0.0

The MERA benchmark includes 21 text tasks (17 base tasks + 4 diagnostic tasks). See the task-table for a complete list.

Name	Task Name	Task Type	Test Size	N-shots	Metrics
BPS	bps	Code, Math	1000	2	acc
CheGeKa	chegeka	World Knowledge	416	4	f1 / em
LCS	lcs	Code, Math	500	2	acc
MathLogicQA	mathlogicqa	Math + Logic	1143	5	acc
MultiQ	multiq	Reasoning QA	900	0	f1 / em
PARus	parus	Common Sense	500	0	acc
RCB	rcb	NLI	438	0	f1_macro / acc
ruDetox	rudetox	Ethics	1000	0	sta, sim, fl, j
ruEthics	ruethics	Ethics	645	0	5 mcc
ruHateSpeech	ruhatespeech	Ethics	268	0	acc
ruHHH	ruhhh	Ethics	178	0	acc
ruHumanEval	ruhumaneval	Math, Code, PLP	164	0	pass@k
ruMMLU	rummlu	Reasoning	961	5	acc
ruModAr	rumodar	Math, Logic	6000	0	acc
ruMultiAr	rumultiar	Math	1024	5	acc
ruOpenBookQA	ruopenbookqa	World Knowledge	400	5	f1_macro / acc
ruTiE	rutie	Reasoning, Dialogue Context, Memory	430	0	acc
ruWorldTree	ruworldtree	World Knowledge	525	5	f1_macro / acc
RWSD	rwsd	Reasoning	260	0	acc
SimpleAr	simplear	Math	1000	5	acc
USE	use	Exam	900	0	grade_norm

Our aim is to evaluate all the models:

• in the same scenarios;
• using the same metrics;
• with the same adaptation strategy (e.g., prompting);
• allowing for controlled and clear comparisons.

Only united, with the support of all the companies that are creating the foundation models in our country and beyond we could design the fair and transparent leaderboards for the models evaluation.

Our team and partners:

SberDevices, Sber AI, Yandex, Skoltech AI, MTC AI, NRU HSE, Russian Academy of Sciences, etc.

Powered by Aliance AI

Contents

The repository has the following structure:

• examples - the examples of loading and using data.
• humanbenchmarks - materials and code for human evaluation.
• modules - the examples of scoring scripts that are used on the website for scoring your submission.
• lm-evaluation-harness - a framework for few-shot evaluation of language models.

Submit to MERA

• To see the datasets use the HuggingFace datasets interface. See the example of the datasets in the prepared Jupyter Notebook.
• To run your model on the all datasets please use the code of lm-harness. The result of the code is the archive in ZIP format for the submission.
• Register on the website and submit your the ZIP. The results will be available for you privately in the account.

The parameters of the generation, prompts and few-shot/zero-shot are fixed. You can vary them for your own purposes. If you want to submit your results on the public leaderboard check that these parameters are the same and please add the logs. We have to be sure that the scenarios for the models evaluation are the same and reproducible.

We provide thesample submission for you to check the format.

The process of the whole MERA evaluation is described on the Figure:

---

📌 It’s the first text version of the benchmark. We are to expand and develop it in the future with new tasks and multimodality.

Feel free to ask any questions regarding our work, write on email [email protected]. If you have ideas and new tasks feel free to suggest them, it’s important! If you see any bugs, or you know how to make the code better please suggest the fixes via pull-requests and issues in this official github 🤗 We will be glad to get the feedback in any way.