Adding Moorcheh (VectorDB Embedding) to MTEB Benchmark

[bhavanipot]

Hello MTEB Team,

We’re working on integrating our model, Moorcheh, into the MTEB benchmark. Moorcheh is a vector database with its own re-embedding method which is intertwined with its retrieval system, making it challenging to return embeddings in order to perform the exact MTEB tasks for the benchmark.

We were wondering if there were alternative methods or abilities to perform the tasks without returning the explicit embeddings so that Moorcheh can be a part of the benchmark (leaderboard), while still ensuring fair and meaningful evaluation.

You can visit our GitHub repository and website for reference.

We would greatly appreciate any guidance on how to proceed.

Thank you,
The Moorcheh Team

[KennethEnevoldsen]

Great that you reached out! Sounds like you want to evaluate the model for retrieval only? If so, then I think that it might be possible once we have fixed this issue: #2728

[bhavanipot]

Thanks for pointing us to issue #2728 — it looks like exactly what we need in order to proceed. Do you have a rough estimate of when this might be implemented or available for testing?

Best regards,
The Moorcheh Team

[KennethEnevoldsen]

I believe v2 is about 1-2 months away. I am unsure of this feature will make it in though. It is a feature I would love to have though so def. on the priority list.