Possible to add a model to the leadboard locally ?

[aladeen57]

Hello,

I ran the MTEB evaluation on the MTEB multilingual V1 benchmark on the retrival task for the model I have developped using this code :

# Load model and tasks
model = mteb.get_model(<PATH_TO_MY_MODEL>")

# Define tasks
benchmark = mteb.get_benchmark("MTEB(Multilingual, v1)")
tasks = [task for task in benchmark.tasks if task.metadata.type == "Retrieval"]
evaluation = mteb.MTEB(tasks=tasks)

# Run evaluation
evaluation.run(model, output_folder="<OUTPUT_PATH>")

I'd like to compare its results to existing models through the leadboard or a dataframe replicating it.
I tried using your documentation to do this but have an error, here is the code :

benchmark = mteb.get_benchmark("MTEB(Multilingual, v1)")
tasks = [task for task in benchmark.tasks if task.metadata.type == "Retrieval"]

model_names = [
    "intfloat/multilingual-e5-large",
    "<PATH_TO_MY_MODEL>"
]
models = [mteb.get_model_meta(name) for name in model_names]

results = mteb.load_results(models=models, tasks=tasks)

df = results_to_dataframe(results)

But I'm facing an issue, here is the error :

WARNING:mteb.models.overview:Failed to extract metadata from model: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '<PATH_TO_MY_MODEL>'. Use `repo_type` argument if needed..

Thanks in advance !

[Samoed]

Currently, you can't do that, but we have developing PR, that will make this possible #2454

[aladeen57]

Thank you @KennethEnevoldsen , I have tried the second option you gave me but the last line of code gave me this error :

TypeError: unsupported operand type(s) for +=: 'BenchmarkResults' and 'BenchmarkResults'

[KennethEnevoldsen]

The second part was an ideal situation (but it doesn't work). The first part does

[aladeen57]

Sorry @KennethEnevoldsen I didn't get the second part was an ideal situation.

I tried the first part and it worked (but had to be carefull with the naming of the result's folder and had to change the value of the key "name" in the model_meta.json to make it work).

Altho I have an issue, the model "intfloat/multilingual-e5-large" seems to only have results for three datasets (ArguAna, TRECCOVID, WikipediaRetrievalMultilingual) when requesting scores on the benchmark MTEB(Multilingual, v1)") and for Retrieval task only.

Here is my full code :

import mteb
from mteb.task_selection import results_to_dataframe

# Define tasks
benchmark = mteb.get_benchmark("MTEB(Multilingual, v1)")
tasks = [task for task in benchmark.tasks if task.metadata.type == "Retrieval"]

# Define models to compare
model_names = [
    "intfloat/multilingual-e5-large",
    "my_model"
]
models = [mteb.get_model_meta(name) for name in model_names]

# Get results
results = mteb.load_results(download_latest=False, models=models, tasks=tasks)


# Put results in df
df = results_to_dataframe(results)

# Save into excel file
df.to_excel("file.xlsx")

I belive that to replicate it's necessary to use at least one time load_results on "intfloat/multilingual-e5-large" without download_latest=False so it stores the results in here : Path.home() / ".cache" / "mteb" (which I did just a few days ago so I belive I'm uptodate). And then the arg download_latest=False will work, but in my case I only have 3 scores on the three dataset I mentionned.

[KennethEnevoldsen]

Sorry for the late response, @aladeen57.

I think the best solution for you is:

1. download results repo using mteb.load_results()
2. (optional) if you want to add your own models results, then simply add them to the repo
3. load the new results using mteb.load_results(download_latest=False, models=models, tasks=tasks)

As for the problems you experience you can just pass the model_names:

model_names = [
    "intfloat/multilingual-e5-large",
    "my_model"
]

# this forced a specific revision, while results on the leaderboard is merged from multiple revisions
# models = [mteb.get_model_meta(name) for name in model_names]

# so simply:
results = mteb.load_results(download_latest=False, models=models, tasks=tasks)

# to combine the results across revisions:
results = results.join_revisions()

I have been working on a PR (#2542) addressing this. Once it is merged you can simply call

results.to_dataframe(include_revisions=False) # default, this will automatically join revisions

The new PR improved the documentation on loading results.

[KennethEnevoldsen]

PR is in, see the new docs here: https://github.com/embeddings-benchmark/mteb/blob/main/docs/usage/results.md