Unable to reproduce MTEB leaderboard score

[AndreaSottana]

I am using your model Snowflake/snowflake-arctic-embed-m-v1.5 to reproduce the MTEB leaderboard results on the TRECCOVID dataset. Based on the leaderboard, the Normalized Discounted Cumulative Gain @ 10 (nDCG@10) for TRECCOVID should be 84.61. I am getting 39.39.
If you cannot see the model Snowflake/snowflake-arctic-embed-m-v1.5 easily, you should set the task to “Retrieval” only from the leaderboard and the model should be in position 35 from the top.

Below is how I tried to reproduce the results

import mteb
from sentence_transformers import SentenceTransformer

model_name = "Snowflake/snowflake-arctic-embed-m-v1.5"

model = SentenceTransformer(model_name, trust_remote_code=True)
tasks = mteb.get_tasks(tasks=["TRECCOVID"])
evaluation = mteb.MTEB(tasks=tasks)
results = evaluation.run(model, output_folder=f"results/{model_name}")
print(results[0].scores['test'][0]['ndcg_at_10'])

Expected output: 0.8641
Actual output: 0.3939

Model details: "name": "Snowflake/snowflake-arctic-embed-m-v1.5", "revision": "3b5a16eaf17e47bd997da998988dce5877a57092"

System specifications:
MacOS Sonoma 14.6.1 (23G93)
python 3.12.5
mteb 1.14.20