New model submission: "Mean (Task)" score

[unnir]

I would like to include our light-weight to the headboard.

I have evaluated the model using this script:

# Full MTEB Evaluation
import mteb
from sentence_transformers import SentenceTransformer

# Load your model
model_name = "tabularisai/Zip-1"
model = SentenceTransformer(model_name)

# Get the full English benchmark
benchmark = mteb.get_benchmark("MTEB(eng, v2)")
print(f"Running full benchmark with {len(benchmark.tasks)} tasks")

# Run the evaluation
print("Starting evaluation - this will take several hours...")
output_folder = f"results/{model_name}"
evaluation = mteb.MTEB(tasks=benchmark.tasks)
results = evaluation.run(model, output_folder=output_folder, batch_size=32768)

However, in the results variable we have different bechmarks and tasks.
My question is how to get the "Mean (Task)" score, which is showed on the main bubble plot (y-axis)?
Also, is my script the right way to go?

[Samoed]

Hi! You can download results using this documentation and after that to average their scores https://github.com/embeddings-benchmark/mteb/blob/main/docs/usage/results.md

Also, is my script the right way to go?

If you don't need additional configuration, then it will be fine, but preferably to run mteb.get_model(model_name) when your model is added to MTEB with metadata.